search

TUW-GEO / geopathfinder

Querying and searching data on the file system
MIT License
0 stars 2 forks source link

Logfile directory level #26

Open cnavacch opened 2 years ago

cnavacch commented 2 years ago

In regard to the yeoda_path convention, the "logfiles" folder is at the same level as the "data_version" level at the moment. The advantage is that the level below "data_version" solely consists of sub-directories in a spatial context, e.g. different Equi7 continents.

However, in the context of job file logging under "logfiles", which are bound to a certain data version, we have the issue that they are hierarchically not connected with the different data versions anymore. This means if someone wants to move data produced with a specific version somewhere else, then it needs to be assured that the respective log files are also moved.

This issue could be solved by either:

  1. Leaving as is - results in problem/inconsistency mentioned above.
  2. Creating a new level below "data_version", e.g. consisting of "datasets" and "logfiles". This would thematically be the best separation, but introduces one more folder level.
  3. Move the "logfiles" folder below "data_version", which would cause an inconsistency with the spatial context of this folder level.
bbauerma commented 2 years ago

apart from being "ugly" having the folder "logfiles" at the grid level, is there any technical/logical backdraft to it?

cnavacch commented 2 years ago

In my view there is no backdraft to it, except that it looks more ugly and that one can't collect subgrid folders that easily anymore (one needs to exclude the "logfiles" folder). But in terms of data consistency, going for option 3 would be the right way to go, I guess.

bbauerma commented 2 years ago

CN and BBM just decided to remove "logfiles" etc completely from yeoda_path within geopathfinder. It just deals with geodata and its structure logfiles folders are completely governed by the workers and wrappers