Ginga data files

[pllim]

If I want to develop a plugin that comes with data file(s), e.g., ASCII table(s), what is the best way to do it? Does Ginga have a scheme for distributing associated data files? Thanks!

[ejeschke]

@pllim, I think whatever scheme we come up with for #92 should also handle associated data for plugins. I really want to solve this, but in a way that is clean and does not rely heavily on the changes between versions of python or python package manager du jour (python packaging seems to change as often as the wind).

[pllim]

Now that #169 is merged, should we revisit this?

[ejeschke]

How about adding a data directory that uses the namespace trick?

[pllim]

Where should this data reside? I can give that a try and see.

[ejeschke]

How about ginga.data?

Don't forget to add it to setup.py. For now it can probably go into the packages list.

[ejeschke]

Hmm, rethinking this. It might need to go in package_data, probably following the way that ginga examples are now installed. At least, that's most likely how you would specify the data files in your setup.py that was installing. I bet @embray would know the answer to this.

[pllim]

We can always utilize astropy.utils.data.get_pkg_data_filename() but I don't know if that will work with shared namespace, nor the best place to place the data directory in this situation.

[ejeschke]

Let's use ginga.data. Try to add it to your package_data in your install script for your plugin and see if you can install data files.

[pllim]

That works. I put my data file under mymodule/ginga/data directory. And added {'ginga': ['data/myfile.txt']} to my package_data in setup.py.

The catch is that I need the latest astropy to find it because I need the new package='ginga' keyword in get_pkg_data_filename(). Otherwise, it goes into "search online" mode and gives me a weird "cannot find _bsddb" error.

[pllim]

@ejeschke , if you are okay with what I reported above, you can close this issue.

[ejeschke]

I think this is ok. You can also (since you need an __init__.py in the directory, right?) do something like:

mydatafile = "specialflags.dat"
from ginga.data import __file__
datadir, _x = os.path.split(__file__)
myfile = os.path.join(datadir, mydatafile)

[pllim]

No, I don't have __init__.py in the data directory, since the data directory is not technically a sub-module.

[ejeschke]

Ok, well you could add an init file or use a parent directory, e.g.

mydatafile = "specialflags.dat"
from ginga import __file__
parentdir, _x = os.path.split(__file__)
myfile = os.path.join(parentdir, 'data', mydatafile)

[pllim]

Since Ginga is already an affiliated package of Astropy, it makes more sense to me to use something from astropy.utils.data. Using that also enables the feature for Ginga to auto-download data files as needed from the web, should you choose to do that in the future.

[ejeschke]

@pllim, if you want to use the astropy data directory function for your plugin data that would be a good idea (especially if your plugin needs astropy). I think its not a bad idea to have a data directory inside the ginga installation path though, at least for data files used for ginga itself, so I think it makes sense to have a ginga.data (or something similar). While astropy is a core requirement for the reference viewer (as several plugins depend on it) it is not a requirement for the viewer widget and it might be a bit odd to require astropy just to to fetch a data file.

[pllim]

Let me clarify: My data file is indeed in ginga.data but I am using Astropy mechanism to retrieve it. The way I do it will not introduce Astropy dependency on the viewer itself, but rather on the specific plugin that requires the data file.

[ejeschke]

Sounds good. Do you have a PR for this? I guess it is just a small setup.py change. Wonder if we should put an __init__.py file in the data directory anyway so git will track the directory and create it on installation.

[pllim]

@ejeschke

Do you have a PR for this?

A PR won't be necessary until you have actual data files, right? Anyways, if you do want me to submit a PR for #162, it comes with a data file, so this would be part of that PR.

[ejeschke]

I think setup.py will need to be modified, at a minimum. Also, it would be nice if the directory were created when the user installs ginga.

[pllim]

Update: I don't know why, but get_pkg_data_filename now cannot find the data file anymore. It was working for the past few days. I'll investigate.

[pllim]

Now that I look at the Astropy codes, I am not sure why it was working at all. Perhaps it was a fluke. Does not seem trivial to update astropy.utils.data.get_pkg_data...() functions to support shared namespace. I will attempt to write something specific to our needs.

[ejeschke]

I would not expect that astropy would find files stored in other package areas. But maybe there is an astropy function to store a file in the astropy area, or wherever it caches such downloads. In that case you would need to use that function to install your data file.

But I think it makes more sense probably to keep the data files close to the plugin. If ginga.data becomes a namespace, then you can simply install a data file along with your own plugin.

[pllim]

I think there is a workaround in #176 #183 (see nstools.py)

[pllim]

No go.