executablebooks / jupyter-cache

A defined interface for working with a cache of executed jupyter notebooks
https://jupyter-cache.readthedocs.io
MIT License
49 stars 14 forks source link

ENH: Support for Dependant files required for Execution runtimes [Julia] #67

Open mmcky opened 3 years ago

mmcky commented 3 years ago

Is your feature request related to a problem? Please describe.

For Julia based projects some authors have been requesting the ability to include julia files such as Manifest.toml and Project.toml to get fine grain control over the julia packages used. This control seems a lot more important in the julia context than python. Sometimes these files can be nested in directories to change the behaviour of how julia imports packages and continuous_time directory would use the local Project.toml and Manifest.toml files in preference to the global ones at the root level of lectures. :

/lectures
    Project.toml
    Manifest.toml
    /dynamic_programming
       # source files
   /continuous_time
       Project.toml
       Manifest.toml
      # source files

https://pkgdocs.julialang.org/v1/toml-files/

Looking at the docs:

Store external assets: Notebooks being executed often require external assets: importing scripts/data/etc. These are prepared by the users.

@chrisjsewell I had a look at the python api in the docs but couldn't see how files get added as dependencies. We could use this mechanism after detecting a julia kernel to look for some set files such as the ones above and copy them into jupyter-cache for execution.

Describe the solution you'd like

We can look at adding support to automatically copy these files (based on the folder structure) when a julia kernel is detected.

Describe alternatives you've considered

None

jlperla commented 3 years ago

Thanks. Yeah, as this is something essential for Julia I think it is best if it is automatic. In practice every Julia project would or should have them.

AakashGfude commented 3 years ago

@mmcky @jlperla after looking at the codebase. It seems that the execution happens in the source directory itself. So, this issue is already handled. Unless you are executing in a temp directory by turning on this option:

execute:
   run_in_temp: true

In this case, execution happens in this temp directory and dependant files are not copied there.

jlperla commented 3 years ago

Thanks @AakashGfude .We will follow whatever you guys tell us to do. I also really appreciate want to make sure we can use the same devops to whatever extent possible with the python books.

Just keep in mind that we want the separate Project.toml files at the top level of the source tree and at a few sub-trees as well.

And that it needs to execute with the those same relative paths. And when we generate the notebook repo we need all of those as well.

mmcky commented 3 years ago

And when we generate the notebook repo we need all of those as well.

We will need to add this to sphinx-tojupyter as a builder step in https://github.com/QuantEcon/sphinx-tojupyter/issues/33