Directory structure not easy to grasp

[unndreay]

I wonder if the structure is compulsory for us. Otherwise, I propose to switch from the existing structure of the scheme

{experiment name}/00#_{processing step}/{UseCase name}

to the following

usecases/{UseCase name}/00#_{processing step}

Why?

• Strictly hierarchical, easy to understand
• Each UseCase is calculated separately during development and so the directories should be isolated as well. Data for each UseCase is grouped and can be copied or deleted easily without hopping from folder to folder.
• Path operations in the code are easier to understand.
• The structure follows a simple principle: Python scripts that can be run and that manipulate data write files only into directories down the hierarchy, never on the same level or above (for reading it's different)

Additionally, consider:

• that the name parts "data" and "results" are not relevant. It puts a difference in the user's mind which I don't see. We could omit that without loosing anything. It will make the path statements in the code more readable.
• that the content of both 004_results_template and 001_data_raw seem identical to me across all the experiments and UseCases. We better move this (kind of) input data to a folder outside the results context
• that UseCase names include the experiment name even now. Why not leave the structure flat by using a general experiments or UseCases or scenarios usecases folder?
• that UseCase names in the run scripts file names won't be necessary anymore because of the isolation of the UseCases (easier in the code) (already done)

• 96

• The comparison folder is the actual result of all oemoflex activities. Also, data from other models come into play (in contrast to all the other folders). Putting it on a top hierarchy level would reflect both facts. This is analog to data: whereas data is the input from the project partners, comparison is the output to them.

This leads us to the following structure:

[usecases]
    - [FlexMex_1_1]
        - [001_preprocessed]
    - [002_optimization]
    - [003_postprocessed]
    - infer.py
    - preprocessing.py
    - optimization.py
    - postprocessing.py
    - runall.py
    - [FlexMex_1_2]
        - [001_preprocessed]
        - ...
    ...
    - [FlexMex_2_1]
    - [FlexMex_2_2]
    ...

[meta-data]                (see #96)
    - mapping-input-scalars.yaml
    - mapping-input-timeseries.yaml
    - mapping-output-scalars.yaml
    - mapping-output-timeseries.yaml
    - [output_template]    ('004_results_template' before)
        - Scalars.csv
        - (dir structure for the timeseries)

[data]                     ('001_data_raw' before)
    - [In]
    - [Out]

[comparison]     ('004_comparison' before)
    - compare_scalars.py
    - join_results.py

[oemoflex]
    - oemof-tabular-settings.yml

EDIT 2020/10/16: New names for some dirs. Comparison moved. #96 added. EDIT 2020/11/16: Move oemoflex-tabular-settings.yml.

[jnnr]

Thanks for these ideas. If a restructuring helps getting an overview, we should do it. Let us however first finish the remaining use cases and make the ideas here more precise before we change something.

[unndreay]

With one script file per use case, the 'usecases' dir hierarchy becomes even easier:

[usecases]
    - [FlexMex_1_1]
        - [001_preprocessed]
    - [002_optimization]
    - [003_postprocessed]
    - [FlexMex_1_2]
        - [001_preprocessed]
        - ...
    ...
    - [FlexMex_2_1]
    - [FlexMex_2_2]
    ...
    - FlexMex_1_1.py
    - FlexMex_1_2.py
    ...
    - FlexMex_2_1.py
    - FlexMex_2_2.py

To delete a run result, we just have to simply delete one [FlexMex_x_y] dir, without touching the script files. The same is true for keeping a copy of a run. We just have to rename one folder and it won't be touched again by later runs.

[jnnr]

We included the thoughts discussed here in #122, therefore we close this issue.