search

edmcman / ocaml-jupyter-binder-environment

An environment for running the ocaml-jupyter kernel in binder
4 stars 2 forks source link

Lots of missing packages, some notebooks are broken #1

Closed Naereen closed 3 years ago

Naereen commented 3 years ago

Hello there @edmcman and @akabe and @sei-eschwartz, (long time user of ocaml-jupyter here)

I just tried again the binder environment based on this Docker image, and all the notebooks. Lots of missing packages, some notebooks are broken, here are the errors I got, with screenshots and the detailed list of notebooks:

plplot.ipynb

# #thread
# #require "plplot,jupyter.notebook"
No such package: plplot

Capture d’écran_2021-02-23_01-18-43

introduction.ipynb

# #require "jupyter-archimedes"
No such package: jupyter-archimedes

Capture d’écran_2021-02-23_01-20-21 Apparently, it's enough to add the jupyter-archimedes package to opam, as running

# Sys.command "opam install -y cairo2 archimedes jupyter jupyter-archimedes";;
NOTE] It seems you have not updated your repositories for a while. Consider updating them with:
       opam update

[NOTE] Package jupyter is already installed (current version is 2.7.3).
[NOTE] Package archimedes is already installed (current version is 0.4.19).
[NOTE] Package cairo2 is already installed (current version is 0.6.2).

The following actions will be performed:
  - install jupyter-archimedes 2.7.2

<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[jupyter-archimedes.2.7.2] downloaded from https://github.com/akabe/ocaml-jupyter/releases/download/v2.7.2/jupyter-v2.7.2.tbz

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
-> installed jupyter-archimedes.2.7.2
Done.

- : int = 0

worked from the notebook itself!

slap_two_layer_neural_network

# #require "core";;
# #require "slap";;
# #require "slap.top";;
# #require "slap.ppx";;
# #require "jupyter-archimedes" ;;
No such package: core
No such package: slap
No such package: slap.top
No such package: slap.ppx
..
Cannot find file archimedes_graphics.cma.
..

Again, it seems that it's easy to fix:

# Sys.command "opam install core slap";;
... it takes a while, like 10 minutes !
... it broke after
-> installed core_kernel.v0.11.1

So it should just be a matter of adding these two packages core and slap in the Docker file... I don't know why it broke, but here's the full output: failed_install_of_Core.txt

simple_image_filtering.ipynb

File "[3]", line 4, characters 65-70: Error: The function applied to this argument has type w:int -> h:int -> Cairo.Surface.t This argument cannot be applied without label 3: let pixels = reshape_1 (genarray_of_array3 rgba) (height width 4) in 4: Cairo.Image.(create_for_data8 ~stride:(width * 4) pixels RGB24 width height)

- it's easy to fix, just add `~w:width ~h:height` in this line.
- After that, I had to remove `~output:` in line `Cairo.PNG.writ...` but then it gave this error:

File "[7]", line 8, characters 28-49: Error: This expression has type string -> unit but an expression was expected of type Cairo.Surface.t 7: let b = Buffer.create 256 in 8: Cairo.PNG.write_to_stream __(Buffer.add_string b)__ (create_rgba_image rgba) ;

Indeed the signature of `Cairo.PNG.write_to_stream` changed. To conclude, these two lines had to change
```diff
-   Cairo.Image.(create_for_data8 ~stride:(width * 4) pixels RGB24 width height)
+   Cairo.Image.(create_for_data8 ~stride:(width * 4) pixels RGB24 ~w:width ~h:height)
...
- Cairo.PNG.write_to_stream ~output:(Buffer.add_string b) (create_rgba_image rgba) (Buffer.add_string b) ;
+ Cairo.PNG.write_to_stream (create_rgba_image rgba) ;

The next of ~output:(...) should remove the label, it's not available in Cairo library anymore.

-   Cairo.PNG.write_to_stream surface ~output:(Buffer.add_string b) ;
+   Cairo.PNG.write_to_stream surface (Buffer.add_string b) ;

random_dataset_generation.ipynb

Again same problem, I don't include screenshots:

No such package: gsl

I tried Sys.command "opam install gsl";; which couldn't compile as the linux package libgsl-dev is not installed. I think it should be easy to add a apt install libgsl-dev to the Dockerfile.

formant_estimation_by_AR.ipynb

I thought it will have the same problem with Core module unavailable, but no it worked! However, loading FFTW3 failed here too, like in the next notebook.

In the next code cell, there is a Sys.command ... calling ffmpeg which is unavailable in the Docker image! Again I think it should be easy to add a apt install ffmpeg to the Dockerfile.

The next cells are broken but I think correct, they just don't have the Fftw3 module and the sound file, as ffpmeg was unavaible.

fftw3_example.ipynb

# #require "fftw3";;
No such package: fftw3

I could ask to install it using Sys.command "opam install fftw3 again, but the compilation failed, as the system (linux) package bfftw3-dev is missing. Again I think it should be easy to add a apt install bfftw3-dev to the Dockerfile. Running Sys.command "apt search bfftw3-dev" gave

libfftw3-dev/groovy 3.3.8-2ubuntu6 amd64
  Library for computing Fast Fourier Transforms - development

so it's indeed available in the apt repository for this Docker image.

owl.ipynb

Again, No such package: owl. And running Sys.command "opam install owl";; took a long time, apparently it works for most steps but

[ERROR] The compilation of conf-openblas failed at "/usr/bin/sh -exc cc $CFLAGS test.c -lopenblas".
... loooong time
The packages you requested declare the following system dependencies. Please make sure they are installed before retrying:
    liblapacke-dev libopenblas-dev

Therefore liblapacke-dev and libopenblas-dev linux/Debian package are missing.

gaussian_random_walk.ipynb

Same problem with gsl package not present, and libgsl-dev not present on the system.

cohttp_lwt_DuckDuckGoAPI.ipynb

No such package: cohttp.lwt, but Sys.command "opam install cohttp";; worked!

Then open Lwt.Infix ;; failed, but it's easy to fix it: just run #require "lwt";; the cell before.

Later, Lwt_main was missing, so I ran Sys.command "opan install cohttp-lwt cohttp-lwt-unix (which takes a long time!) but then #require "Cohttp_lwt";; failed (missing). I have no idea, even after reading the OCaml-Cohttp GitHub page and its page on OPAM...

It's possible that the OPAM packages index on the Docker image are outdated, as the system gave me a warning, and I saw that the versions of cohttp which were being installed were 2.4.0 but it's 3.0.0 on OPAM... So, I tried to run Sys.command "opam update", but couldn't upgrade to more recent version of Cohttp:

# Sys.command "opam install cohttp.3.0.0 cohttp-async.3.0.0 lwt cohttp-lwt.3.0.0 cohttp-lwt-unix.3.0.0";;
The following dependencies couldn't be met:
  - cohttp-lwt -> ocaml >= 4.08
      base of this switch (use `--unlock-base' to force)
No solution found, exiting
- : int = 20

cohttp_async_DuckDuckGoAPI.ipynb

Same problem but with cohttp-async. I didn't try.

Conclusion

Adding these should solve all the issues detailed above, except the Cohttp_lwt problem.

Cheers, regards from France, @NaereenHello there @edmcman and @akabe and @sei-eschwartz, (long time user of ocaml-jupyter here)

I just tried again the binder environment based on this Docker image, and all the notebooks. Lots of missing packages, some notebooks are broken, here are the errors I got, with screenshots and the detailed list of notebooks:

plplot.ipynb

# #thread
# #require "plplot,jupyter.notebook"
No such package: plplot

Capture d’écran_2021-02-23_01-18-43

introduction.ipynb

# #require "jupyter-archimedes"
No such package: jupyter-archimedes

Capture d’écran_2021-02-23_01-20-21 Apparently, it's enough to add the jupyter-archimedes package to opam, as running

# Sys.command "opam install -y cairo2 archimedes jupyter jupyter-archimedes";;
NOTE] It seems you have not updated your repositories for a while. Consider updating them with:
       opam update

[NOTE] Package jupyter is already installed (current version is 2.7.3).
[NOTE] Package archimedes is already installed (current version is 0.4.19).
[NOTE] Package cairo2 is already installed (current version is 0.6.2).

The following actions will be performed:
  - install jupyter-archimedes 2.7.2

<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[jupyter-archimedes.2.7.2] downloaded from https://github.com/akabe/ocaml-jupyter/releases/download/v2.7.2/jupyter-v2.7.2.tbz

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
-> installed jupyter-archimedes.2.7.2
Done.

- : int = 0

worked from the notebook itself!

slap_two_layer_neural_network

# #require "core";;
# #require "slap";;
# #require "slap.top";;
# #require "slap.ppx";;
# #require "jupyter-archimedes" ;;
No such package: core
No such package: slap
No such package: slap.top
No such package: slap.ppx
..
Cannot find file archimedes_graphics.cma.
..

Again, it seems that it's easy to fix:

# Sys.command "opam install core slap";;
... it takes a while, like 10 minutes !
... it broke after
-> installed core_kernel.v0.11.1

So it should just be a matter of adding these two packages core and slap in the Docker file... I don't know why it broke, but here's the full output: failed_install_of_Core.txt

simple_image_filtering.ipynb

File "[3]", line 4, characters 65-70: Error: The function applied to this argument has type w:int -> h:int -> Cairo.Surface.t This argument cannot be applied without label 3: let pixels = reshape_1 (genarray_of_array3 rgba) (height width 4) in 4: Cairo.Image.(create_for_data8 ~stride:(width * 4) pixels RGB24 width height)

- it's easy to fix, just add `~w:width ~h:height` in this line.
- After that, I had to remove `~output:` in line `Cairo.PNG.writ...` but then it gave this error:

File "[7]", line 8, characters 28-49: Error: This expression has type string -> unit but an expression was expected of type Cairo.Surface.t 7: let b = Buffer.create 256 in 8: Cairo.PNG.write_to_stream __(Buffer.add_string b)__ (create_rgba_image rgba) ;

Indeed the signature of `Cairo.PNG.write_to_stream` changed. To conclude, these two lines had to change
```diff
-   Cairo.Image.(create_for_data8 ~stride:(width * 4) pixels RGB24 width height)
+   Cairo.Image.(create_for_data8 ~stride:(width * 4) pixels RGB24 ~w:width ~h:height)
...
- Cairo.PNG.write_to_stream ~output:(Buffer.add_string b) (create_rgba_image rgba) (Buffer.add_string b) ;
+ Cairo.PNG.write_to_stream (create_rgba_image rgba) ;

The next of ~output:(...) should remove the label, it's not available in Cairo library anymore.

-   Cairo.PNG.write_to_stream surface ~output:(Buffer.add_string b) ;
+   Cairo.PNG.write_to_stream surface (Buffer.add_string b) ;

random_dataset_generation.ipynb

Again same problem, I don't include screenshots:

No such package: gsl

I tried Sys.command "opam install gsl";; which couldn't compile as the linux package libgsl-dev is not installed. I think it should be easy to add a apt install libgsl-dev to the Dockerfile.

formant_estimation_by_AR.ipynb

I thought it will have the same problem with Core module unavailable, but no it worked! However, loading FFTW3 failed here too, like in the next notebook.

In the next code cell, there is a Sys.command ... calling ffmpeg which is unavailable in the Docker image! Again I think it should be easy to add a apt install ffmpeg to the Dockerfile.

The next cells are broken but I think correct, they just don't have the Fftw3 module and the sound file, as ffpmeg was unavaible.

fftw3_example.ipynb

# #require "fftw3";;
No such package: fftw3

I could ask to install it using Sys.command "opam install fftw3 again, but the compilation failed, as the system (linux) package bfftw3-dev is missing. Again I think it should be easy to add a apt install bfftw3-dev to the Dockerfile. Running Sys.command "apt search bfftw3-dev" gave

libfftw3-dev/groovy 3.3.8-2ubuntu6 amd64
  Library for computing Fast Fourier Transforms - development

so it's indeed available in the apt repository for this Docker image.

owl.ipynb

Again, No such package: owl. And running Sys.command "opam install owl";; took a long time, apparently it works for most steps but

[ERROR] The compilation of conf-openblas failed at "/usr/bin/sh -exc cc $CFLAGS test.c -lopenblas".
... loooong time
The packages you requested declare the following system dependencies. Please make sure they are installed before retrying:
    liblapacke-dev libopenblas-dev

Therefore liblapacke-dev and libopenblas-dev linux/Debian package are missing.

gaussian_random_walk.ipynb

Same problem with gsl package not present, and libgsl-dev not present on the system.

cohttp_lwt_DuckDuckGoAPI.ipynb

No such package: cohttp.lwt, but Sys.command "opam install cohttp";; worked!

Then open Lwt.Infix ;; failed, but it's easy to fix it: just run #require "lwt";; the cell before.

Later, Lwt_main was missing, so I ran Sys.command "opan install cohttp-lwt cohttp-lwt-unix (which takes a long time!) but then #require "Cohttp_lwt";; failed (missing). I have no idea, even after reading the OCaml-Cohttp GitHub page and its page on OPAM...

It's possible that the OPAM packages index on the Docker image are outdated, as the system gave me a warning, and I saw that the versions of cohttp which were being installed were 2.4.0 but it's 3.0.0 on OPAM... So, I tried to run Sys.command "opam update", but couldn't upgrade to more recent version of Cohttp:

# Sys.command "opam install cohttp.3.0.0 cohttp-async.3.0.0 lwt cohttp-lwt.3.0.0 cohttp-lwt-unix.3.0.0";;
The following dependencies couldn't be met:
  - cohttp-lwt -> ocaml >= 4.08
      base of this switch (use `--unlock-base' to force)
No solution found, exiting
- : int = 20

cohttp_async_DuckDuckGoAPI.ipynb

Same problem but with cohttp-async. I didn't try.

Conclusion

Adding these should solve all the issues detailed above, except the Cohttp_lwt problem.

Cheers, regards from France, @Naereen

edmcman commented 3 years ago

Adding these packages would be relatively easy. But the better solution is probably to base the docker image on one of these Docker images which has the dependencies already. I vaguely remember running into a problem when I tried this originally. Perhaps it is time to try it again.

edmcman commented 3 years ago

It looks like the python in the datascience Docker image might be too old to use pip anymore...

Waiting for build to start...
Picked Git content provider.
Cloning into '/tmp/repo2dockerzjy5pyb9'...
HEAD is now at af9e650 Is this newer?
Using DockerBuildPack builder
Step 1/4 : FROM akabe/ocaml-jupyter-datascience:debian_ocaml4.05.0
 ---> 0dc642ff8666e...
Step 2/4 : RUN sudo pip3 install --upgrade pip
 ---> Running in 53ecb9670f57
Collecting pip
  Downloading https://files.pythonhosted.org/packages/5c/e0/be401c003291b56efc55aeba6a80ab790d3d4cece2778288d65323009420/pip-19.1.1-py2.py3-none-any.whl (1.4MB)
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
Successfully installed pip-19.1.1
Removing intermediate container 53ecb9670f57
 ---> 61fe40a721fa
Step 3/4 : RUN sudo pip3 install nbgitpuller
 ---> Running in ecb1c83f1509
DEPRECATION: Python 3.4 support has been deprecated. pip 19.1 will be the last one supporting it. Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429).
Collecting nbgitpuller
  Downloading https://files.pythonhosted.org/packages/7b/5b/f7e0bbfad992e1b438cd0ae648185b7d62c4e02e9b9902857397275bfe27/nbgitpuller-0.9.0-py2.py3-none-any.whl
Collecting tornado (from nbgitpuller)
  Downloading https://files.pythonhosted.org/packages/e6/78/6e7b5af12c12bdf38ca9bfe863fcaf53dc10430a312d0324e76c1e5ca426/tornado-5.1.1.tar.gz (516kB)
    ERROR: Complete output from command python setup.py egg_info:
    ERROR: Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ImportError: No module named 'setuptools'
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-c_wh9eam/tornado/
Removing intermediate container ecb1c83f1509
The command '/bin/sh -c sudo pip3 install nbgitpuller' returned a non-zero code: 1
Naereen commented 3 years ago

Thanks for the reply. I won't have more time to give to this issue, I tried all the notebooks last night and spend already quite some time on this.

If you fix it, good, otherwise too bad.

Maybe the problem is just the binder link on https://github.com/akabe/ocaml-jupyter/#running-ocaml-jupyter-on-binder, which does not load the correct (most up-to-date) Docker version? I don't know.

edmcman commented 3 years ago

The problem is that the datascience Docker images are very out of date. I will try to install the packages you suggested in the current version which does not use the datascience images.

Naereen commented 3 years ago

I don't know what's the best plan. @akabe do you think there exists more up-to-date Docker images for OCaml + Jupyter + datascience ? If so, maybe the easiest solution is just to update the link and the mybinder badge and link you give in https://github.com/akabe/ocaml-jupyter/#running-ocaml-jupyter-on-binder Thanks in advance!

edmcman commented 3 years ago

I've been working on this some more on the develop branch. We install many more of the dependencies now, but there are still problems. As @naereen noted, the cohttp examples do not work. I believe this is because the modules are now named cohttp-async instead of cohttp.async. This is simple enough to fix, but doing so would break the notebooks for the datascience image.

The other problem is that adding all these packages makes the binder build process very slow.

I propose to install the packages required only for the introduction notebook, and to change the link on the ocaml-jupyter README to only link to that notebook.

akabe commented 3 years ago

@Naereen @edmcman Sorry for late reply. My docker images (especially datascience) are no loger maintained because it takes time and effort. I agree with @edmcman's opinion:

But the better solution is probably to base the docker image on one of these Docker images which has the dependencies already.

However, I note that installing and updating many packages will consume your time. I recommend to include only popular libraries in a new datascience image if you try to create it.

Naereen commented 3 years ago

Hi @akabe, Thanks for the reply, and I completely understand that this takes time. I've never been convinced by Docker images for a local personal usage, and find them completely useless for data-science in general... except that MyBinder is based on them!! So my issue and question was only opened with the goal of improving/fixing the possible usage of ocaml-jupyter in MyBinder.

I don't want to take the responsibility of keeping the Docker image(s) up-to-date either, sorry... I'm teaching introduction to computer science classes and introduction to OCaml, not advanced data science using OCaml, so for my teaching activities this project is not "vital".

Regards, @Naereen

edmcman commented 3 years ago

I've changed the image so that archimedes is installed and the introduction notebook is now fully functional. I also created a pull request against akabe/ocaml-jupyter#170 to link directly to the introduction notebook. Hopefully all agree this is a reasonable solution for now since it seems like none of us want to devote a lot of time to create a new up-to-date image for the example notebooks.