search

gammapy / gamma-cat

An open data collection and source catalog for gamma-ray astronomy
https://gamma-cat.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
15 stars 17 forks source link

Add year folder in output/data #198

Closed cdeil closed 6 years ago

cdeil commented 6 years ago

I think the folder renames are done, except for one point:

In input/data we have sub-folders for each year, and in output/data we don't.

I think it would be easiest if the input and output folder had the same structure, i.e. we should introduce the year sub-folders in output/data.

From a quick look , what's needed is to remove most of the output filename building code in CollectionConfig and to just take the relative input location also as relative output file location. https://github.com/gammapy/gamma-cat/blob/c028df0d0d06641afc0edef60fe9ed812078055a/gammacat/collection.py#L31

I don't have time for this now. @pdeiml - Either you make a PR for this (with just the scripting changes, not the file renames committed, so that it's easy to review), or I do it some day. Let's here and generally work via Github issues, and just assign ourselves for issues when we start working on them, just in case we both start hacking on gamma-cat again.

pdeiml commented 6 years ago

I agree with you that we should have similar structure in input and output. For me this change needs a bit of time and I don't know when I will have this time but I can do it.

cdeil commented 6 years ago

I noticed that this is a major bug: we don't handle cases with multiple file_id when copying to the output folder, i.e. they are just discarded. Changing output folder / file structure to match input and then simply processing all resources via the index files will resolve this issue.

@pdeiml - Let me know if you have time for a PR to fix this week, or assign to me and I'll do it later this week.

pdeiml commented 6 years ago

I started https://github.com/gammapy/gamma-cat/pull/215 which handles the ideas above. As soon as https://github.com/gammapy/gamma-cat/pull/215 is merged, this issue can be closed.

cdeil commented 6 years ago

I think this issue is fully resolved. Closing See https://github.com/gammapy/gamma-cat/tree/master/output/data