Ouranosinc / PAVICS-e2e-workflow-tests

Test user-level workflow.
Apache License 2.0
0 stars 2 forks source link

Avoid getting a file output flatlist #98

Open matprov opened 2 years ago

matprov commented 2 years ago

Currently every notebook gets outputted to the buildout directory, regardless of the initial folder structure in which they were located. For example, notebooks from /docs/source/notebook-components/ and /docs/source/notebooks/ are sent to the same output directory, which might cause confusion.

The solution would be to keep the initial folder structure. We would then have buildout/notebook-components and buildout/notebooks.

tlvu commented 2 years ago

Another cheap way to achieve this is to have the output file right next to the original file. We already archive the original files so it falls under the same existing archiving rules and if the original files are under different folder structures, it will be preserved as well.

That said, the flat structure was for an easy access on Jenkins when browsing the archived output. Nested structure requires many clicks to go back and forth between the different folders.

I'd suggest this output structure be a toggle so the existing behavior is preserved and the new behavior can be activated when desired.

matprov commented 2 years ago

I would say that simply preserving the input structure for the notebook outputs would be the best way to avoid name clash. Having a toggle for this type of thing is cumbersome for no reason.

tlvu commented 2 years ago

I would say that simply preserving the input structure for the notebook outputs would be the best way to avoid name clash. Having a toggle for this type of thing is cumbersome for no reason.

Yeah I think you might be right. The nested directory structure is slow to navigate but I don't have to dig inside it too often yet. Let's avoid premature optimization.