Companion Jupyter Books - Python and Julia

orectique commented 1 year ago

Hello @jstac,

This PR contains the restructured public DP book repo.

The source code is up to date with the private repo.

The generated figures are under ./julia_version/figures/.

HumphreyYang commented 1 year ago

Hi @orectique,

Many thanks for opening the PR. We might want to remove the built files (pdfs and HTMLs) and cached files after @jstac's review. It is a convention to make the repo light : )

orectique commented 1 year ago

Hello @HumphreyYang,

Thank you for the feedback.

The PDFs are intended to be there for @jstac to link in the book.

I figured I'll leave the generated HTML files there till the CI has been implemented.

HumphreyYang commented 1 year ago

The PDFs are intended to be there for @jstac to link in the book.

I figured I'll leave the generated HTML files there till the CI has been implemented.

Many thanks. Sounds great to me. I have no push right to this repo, so I cannot set up CI as well : )

jstac commented 1 year ago

Many thanks @orectique for taking the lead on this, and @HumphreyYang for helping out.

@mmcky , would you mind giving this a first look over?

orectique commented 1 year ago

Thanks @mmcky for the review. I have implemented the suggested changes.

jstac commented 1 year ago

@mmcky Please let me know when you've had time to look at this PR

mmcky commented 1 year ago

Thanks @orectique I will run this locally as a test. Can you please update the README with instructions on how to use the scripts.

For example, do I just need to use python create_juliabook.py?

mmcky commented 1 year ago

thanks for all your work on this @orectique

Just taking another look this morning and it would be great if we could change a few things to make more self contained "jupyterbooks" as output.

[ ] update figures directory to be included in _static/figures within the jupyter-book itself so that the books are self contained (rather than using relative links that extend beyond the project directory such as (figname="../figures/iid_job_search_0_py.pdf") this should be _static/figures/iid_job_search_0_py.pdf
[ ] remove one nested directory and remove python_version folder and call it python-codebook which contains the files and folders contained in python_version/pythonbook

That would be awesome.

[ ] add an environment.yml file to document what software is required to run both the jb build and the actual content contained in the jb

mmcky commented 1 year ago

add an environment.yml file to document what software is required to run both the jb build and the actual content contained in the jb

If you test with the base quantecon environment then we can just add that to the README as a note.

mmcky commented 1 year ago

@jstac currently this is generating two code-books.

Maybe we should arrange this to be a single jupyter-book such that we have:

a single folder is generated called code-book
a single _toc.yml that makes use of python and julia part/chapter structure
a single _config.yml
two sub-folders for py and jl codebooks

so when the code book is built we would have book-dp1.quantecon.org/py/<page> and the pages are arranged in parts and two separate kernels for python and Julia would be required. Do you see a reason you just want to update a single language (i.e. and keep them as separate projects)?

orectique commented 1 year ago

Hello @mmcky ,

Thank you for the review.

I had left the script instructions in script_guide.md, since QuantEcon/book-dp1 was to be a public facing repo.

mmcky commented 1 year ago

I had left the script instructions in script_guide.md, since QuantEcon/book-dp1 was to be a public facing repo.

Thanks @orectique -- I think we can move that to the README so it is more visible when viewing on GitHub. I don't really see a need to obscure the build instructions as anyone that clones the repo has the tools needed to build the jupyter-book either way. :-)

jstac commented 1 year ago

@jstac currently this is generating two code-books.

Maybe we should arrange this to be a single jupyter-book such that we have:
1. a single folder is generated called `code-book`

2. a single `_toc.yml` that makes use of `python` and `julia` part/chapter structure

3. a single `_config.yml`

4. two sub-folders for `py` and `jl` codebooks
so when the code book is built we would have book-dp1.quantecon.org/py/<page> and the pages are arranged in parts and two separate kernels for python and Julia would be required. Do you see a reason you just want to update a single language (i.e. and keep them as separate projects)?

Good call @mmcky , I agree.

mmcky commented 1 year ago

@orectique happy to link up on Zoom if anything isn't clear to run through next steps.

mmcky commented 1 year ago

@orectique could you let me know your schedule and if you are able to make these changes. John will need this to be up and running by Monday next week. Let me know how I can help.

orectique commented 1 year ago

@mmcky I'll be working on this tonight. Apologies for the delay.

mmcky commented 1 year ago

@mmcky I'll be working on this tonight. Apologies for the delay.

No worries. Thanks @orectique just wanted to check timelines. Appreciate the work.

mmcky commented 1 year ago

@orectique it would be great to review this afternoon if possible. Otherwise I will work on this on Saturday. I am out of the office tomorrow Friday.

orectique commented 1 year ago

sure thing, @mmcky. I'm just running tests now.

mmcky commented 1 year ago

sure thing, @mmcky. I'm just running tests now.

thanks @orectique -- take your time. I can review from around 2pm onwards.

orectique commented 1 year ago

Hi @mmcky. I have compiled the jl and py versions into a single Jupyter book.

The Julia side instantiation has been automated - since the Project.toml file is present, as long as there is a compatible Julia installation on the system, things should work out of the box.

I've chosen to add _julia and _python suffixes to file/chapter names to avoid nesting the files.
For now, I've left the figures/ directory under code-book/ so that the code in John's book do not have references to _static/figures. Happy to update that tonight if the latter is preferred.

mmcky commented 1 year ago

thanks @orectique this is looking nice.

mmcky commented 1 year ago

@orectique I am getting the following error

ERROR: LoadError: ArgumentError: Package JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] is required but does not seem to be installed:

Do you think I just need to install the Package. Never seen this with Julia before. I am running 1.9.2

mmcky commented 1 year ago

I've chosen to add _julia and _python suffixes to file/chapter names to avoid nesting the files.

Roger that. I might move them to use py and jl folders but let's use what you have for now.

orectique commented 1 year ago

ERROR: LoadError: ArgumentError: Package JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] is required but does not seem to be installed:

That is interesting. That's more of an under-the-hood package, isn't it?

I tested this out on my secondary device this morning, to no such error.

orectique commented 1 year ago

I've chosen to add _julia and _python suffixes to file/chapter names to avoid nesting the files.

Roger that. I might move them to use py and jl folders but let's use what you have for now.

I'll change that in a couple of hours.

mmcky commented 1 year ago

thanks @orectique for your work on this. I think we are getting close now. I will pull the latest and do some more local testing.

orectique commented 1 year ago

@mmcky does the JLLWrappers issue persist?

mmcky commented 1 year ago

@mmcky does the JLLWrappers issue persist?

Yup still getting issues with IJulia loading and throwing lots of errors.

How do I instantiate the Manifest.toml manually. Maybe I should try that to setup the Julia side of things and then try jb again

mmcky commented 1 year ago

I have tried

julia --project=lectures --threads auto -e 'using Pkg; Pkg.instantiate();'

but still getting issues with my jb builds. Strange. Julia!

mmcky commented 1 year ago

OK I reinstalled Julia and updated all my packages. Seems to be running. 😮‍💨

mmcky commented 1 year ago

@orectique I am getting a number of issues relating to inconsistent header levels

/Users/mmcky/work/quantecon/book-dp1/code-book/jl/introductions.md:37: WARNING: Non-consecutive header level increase; H1 to H4 [myst.header]

Do you have anytime to fix any # and ## to make sure they follow a consistent structure. Then sphinx won't complain.

mmcky commented 1 year ago

Also could you use julia rather than julia-1.9 for code-cells in the jl collection

mmcky commented 1 year ago

@orectique nice work!

This is looking really nice.

mmcky commented 1 year ago

@orectique Just a few things. Looking at introduction in Julia for example -- it appears there are some missing companion files that need to be in the directory

SystemError: opening file "/Users/mmcky/work/quantecon/book-dp1/code-book/jl/linear_iter.jl": No such file or directory

orectique commented 1 year ago

Hi @mmcky. That shouldn't be happening. Could I please get the traceback for that?

mmcky commented 1 year ago

Seems to be an issue with PyCall

PyError ($(Expr(:escape, :(ccall(#= /Users/mmcky/.julia/packages/PyCall/ilqDX/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'FileNotFoundError'>
FileNotFoundError(2, 'No such file or directory')
  File "/Users/mmcky/.julia/conda/3/lib/python3.9/site-packages/matplotlib/figure.py", line 3019, in savefig
    self.canvas.print_figure(fname, **kwargs)
  File "/Users/mmcky/.julia/conda/3/lib/python3.9/site-packages/matplotlib/backend_bases.py", line 2319, in print_figure
    result = print_method(
  File "/Users/mmcky/.julia/conda/3/lib/python3.9/site-packages/matplotlib/backend_bases.py", line 1648, in wrapper
    return func(*args, **kwargs)
  File "/Users/mmcky/.julia/conda/3/lib/python3.9/site-packages/matplotlib/_api/deprecation.py", line 386, in wrapper
    return func(*inner_args, **inner_kwargs)
  File "/Users/mmcky/.julia/conda/3/lib/python3.9/site-packages/matplotlib/backends/backend_pdf.py", line 2778, in print_pdf
    file = PdfFile(filename, metadata=metadata)
  File "/Users/mmcky/.julia/conda/3/lib/python3.9/site-packages/matplotlib/backends/backend_pdf.py", line 654, in __init__
    fh, opened = cbook.to_filehandle(filename, "wb", return_opened=True)
  File "/Users/mmcky/.julia/conda/3/lib/python3.9/site-packages/matplotlib/cbook/__init__.py", line 451, in to_filehandle
    fh = open(fname, flag, encoding=encoding)

Stacktrace:
  [1] pyerr_check
    @ ~/.julia/packages/PyCall/ilqDX/src/exception.jl:75 [inlined]
  [2] pyerr_check
    @ ~/.julia/packages/PyCall/ilqDX/src/exception.jl:79 [inlined]
  [3] _handle_error(msg::String)
    @ PyCall ~/.julia/packages/PyCall/ilqDX/src/exception.jl:96
  [4] macro expansion
    @ ~/.julia/packages/PyCall/ilqDX/src/exception.jl:110 [inlined]
  [5] #107
    @ ~/.julia/packages/PyCall/ilqDX/src/pyfncall.jl:43 [inlined]
  [6] disable_sigint
    @ ./c.jl:473 [inlined]
  [7] __pycall!
    @ ~/.julia/packages/PyCall/ilqDX/src/pyfncall.jl:42 [inlined]
  [8] _pycall!(ret::PyObject, o::PyObject, args::Tuple{String}, nargs::Int64, kw::Ptr{Nothing})
    @ PyCall ~/.julia/packages/PyCall/ilqDX/src/pyfncall.jl:29
  [9] _pycall!
    @ ~/.julia/packages/PyCall/ilqDX/src/pyfncall.jl:11 [inlined]
 [10] #_#114
    @ ~/.julia/packages/PyCall/ilqDX/src/pyfncall.jl:86 [inlined]
 [11] (::PyObject)(args::String)
    @ PyCall ~/.julia/packages/PyCall/ilqDX/src/pyfncall.jl:86
 [12] fig_v1(model::NamedTuple{(:n, :w_vals, :ϕ, :β, :c), Tuple{Int64, Vector{Float64}, Vector{Float64}, Float64, Float64}}; savefig::Bool, figname::String, fs::Int64)
    @ Main ./In[2]:79
 [13] top-level scope
    @ In[3]:1

mmcky commented 1 year ago

then the missing file error

SystemError: opening file "/Users/mmcky/work/quantecon/book-dp1/code-book/jl/s_approx.jl": No such file or directory

Stacktrace:
  [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
    @ Base ./error.jl:176
  [2] #systemerror#82
    @ ./error.jl:175 [inlined]
  [3] systemerror
    @ ./error.jl:175 [inlined]
  [4] open(fname::String; lock::Bool, read::Nothing, write::Nothing, create::Nothing, truncate::Nothing, append::Nothing)
    @ Base ./iostream.jl:293
  [5] open
    @ ./iostream.jl:275 [inlined]
  [6] open(f::Base.var"#418#419"{String}, args::String; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base ./io.jl:393
  [7] open
    @ ./io.jl:392 [inlined]
  [8] read
    @ ./io.jl:473 [inlined]
  [9] _include(mapexpr::Function, mod::Module, _path::String)
    @ Base ./loading.jl:1959
 [10] include(fname::String)
    @ Base.MainInclude ./client.jl:478
 [11] top-level scope
    @ In[7]:1

mmcky commented 1 year ago

@orectique there are also some missing modules on the python side in my local build such as linear_iter_fig.py with

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[7], line 7
      4 plt.rcParams.update({"text.usetex": True, "font.size": 14})
      5 import numpy as np
----> 7 from linear_iter import x_star, T
     10 fig, ax = plt.subplots()
     12 e = 0.02

ModuleNotFoundError: No module named 'linear_iter'

mmcky commented 1 year ago

@orectique still having issues building this locally.

Can you confirm that you fresh run works well. Then I will go ahead and merge this then setup CI and Publish workflows to do some more testing via GitHub actions.

mmcky commented 1 year ago

@orectique ok I will merge this and try to get it building via CI

orectique commented 1 year ago

@mmcky I'm trying to get it to build in a Linux environment at the moment.

QuantEcon / book-dp1-public-companion

Companion Jupyter Books - Python and Julia #14