refactored nested code for readability

[shimwell]

Description

I spotted a few more nested levels of code and had a go at refactoring it so that the code was a bit more readable. It makes use of all and dictionary comprehension which might make it more complex so perhaps it is not actually any nicer as we trade compactness of code for complexity here. Feel free to reject these PRs I'm just seeing what can be done.

Fixes # (issue)

No open issues, just refactoring

Type of change

more concise code

Please select what type of change this is.

• [x] more concise code

Other changes

• [ ] This change requires a documentation update
• [ ] (If Benchmark) This requires additional data that can be obtained from:
  • Benchmark data 1
  • Benchamrk data 2

Testing

Sorry I've not added tests, happy to do so

Checklist:

• [x] My code follows the style guidelines of this project
• [x] I have performed a self-review of my code
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [x] My changes generate no new warnings
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] General testing
  • [ ] New and existing unit tests pass locally with my changes
  • [ ] Coverage is >80%

[dodu94]

Hi Jonathan, I did a quick test of the new logic and I'm not sure it is doing what is expected as the last level is supposed to be a list of files contained by the last folder:

It seemed also a bit slow (probably not faster than what we have right now). I believe that if we want to refactor this specific part of the code we should use os.walk, it should also be much faster:

[shimwell]

Good spot. Looks like we don't need this PR anymore. Thanks for checking