Colab integration - Githubissues

lorenzoh commented 3 years ago

Ideally, all the documentation pages based off of Jupyter notebooks should have a button to launch them in Colab.

For example, fastai's documentation has such a button:

First step would be a tutorial on how to set up Colab with Julia and install the dependencies.

See this Discourse post that shows how to get Julia running on Colab.

INF800 commented 3 years ago

Hi, I want to take this up.

lorenzoh commented 3 years ago

Cool! Here's what I think needs to be done.

Conversion of regular Julia .ipynb notebooks to Colab notebooks This consists of
- Adding cells with instructions and shell script that installs Julia, packages and the Julia kernel (see the Colab template for an example)
- Updating the metadata of notebook with kernel name and accelerator info so that GPU is used by default
This functionality may be best created as a separate repository (Colab.jl?) that does this conversion; that way the functionality will be easier to reuse but also maintain when the specifics of running Julia on Colab change
Integration with the documentation system (Pollen.jl)

During documentation build, the following needs to happen for every source notebook file
- create a Colab version of the notebook using the above conversion functionality
- generate a Colab link that will auto-open the Colab version that is built to the GitHub Pages branch during doc build
- insert the link with a badge into the documentation html output
Since Pollen.jl is still undocumented, I could take care of this step once the first is done

Let me know if you need any other pointers 👍

Chandu-4444 commented 2 years ago

Hey, I'm interested in doing this. Can I try to do this?

lorenzoh commented 2 years ago

Sure! Let me know if you need any other pointers aside from the above

Chandu-4444 commented 2 years ago

I could come up with this. But is there any way to speed up the initialization process (The time taken to run the first cell). Running this for every time a notebook that has been opened is kind of inefficient? Or is there a workaround?

lorenzoh commented 2 years ago

One though I've had is using PackageCompiler.jl to create a sysimage once that is then saved somewhere and loaded into the Colab session which would allow reducing installation time significantly and almost completely removing compilation overhead.

However, there are a lot of problems that need to be dealt with, and I'm not sure it's worth the effort:

the sysimage needs to be built on Colab since it needs to be compiled for that hardware and software
it needs to be manually rebuilt on every update since Colab doesn't allow automation
the sysimage file needs to be stored somewhere and served (these are easily > 100MB)
I'm not sure if the randomly different GPU hardware you get on Colab would mess with the sysimage.

Chandu-4444 commented 2 years ago

Ok then. I'll add the Launch Colab button to all document pages that are based off a notebook.

Cool! Here's what I think needs to be done.

Conversion of regular Julia .ipynb notebooks to Colab notebooks This consists of

Adding cells with instructions and shell script that installs Julia, packages and the Julia kernel (see the Colab template for an example)

Updating the metadata of notebook with kernel name and accelerator info so that GPU is used by default

This functionality may be best created as a separate repository (Colab.jl?) that does this conversion; that way the functionality will be easier to reuse but also maintain when the specifics of running Julia on Colab change

Integration with the documentation system (Pollen.jl) During documentation build, the following needs to happen for every source notebook file

create a Colab version of the notebook using the above conversion functionality

generate a Colab link that will auto-open the Colab version that is built to the GitHub Pages branch during doc build

insert the link with a badge into the documentation html output

Since Pollen.jl is still undocumented, I could take care of this step once the first is done

Let me know if you need any other pointers +1

Sure! I started working on the first one. Is it possible to update the metadata of a notebook file and add the extra Julia installation to the start of the notebook as soon as the user clicks on Launch Colab? Or is it better to manually add this information to all notebook files and store them in a separate repo (Colab.jl) and we can link the corresponding notebook from Colab.jl as soon as a user clicks Launch Colab in FastAI documentation page.

lorenzoh commented 2 years ago

First part is writing a program that converts a regular Julia notebook to one that can be run on Colab, i.e. has the setup cells and metadata to select the GPU. Should be independent of the buttons.

This code should probably live in a separate repo, but the notebook files themselves will be built during the documentation building step in FastAI.jl's CI and stored on the gh-pages branch.

Chandu-4444 commented 2 years ago

First part is writing a program that converts a regular Julia notebook to one that can be run on Colab, i.e. has the setup cells and metadata to select the GPU. Should be independent of the buttons.

I'll first try writing that conversion script. Thanks 👍

Chandu-4444 commented 2 years ago

First part is writing a program that converts a regular Julia notebook to one that can be run on Colab, i.e. has the setup cells and metadata to select the GPU. Should be independent of the buttons.

I'll first try writing that conversion script. Thanks +1

Hey, I made a repo that contains the code for converting a normal ipynb notebook to the one that can run on Colab. https://github.com/Chandu-4444/Colab.jl This is a sample one, there are few improvements to be made (Specifying the output file location etc).

Please look at the repo and let me know the next step.

Chandu-4444 commented 2 years ago

This code should probably live in a separate repo, but the notebook files themselves will be built during the documentation building step in FastAI.jl's CI and stored on the gh-pages branch.

I have an idea to do this. Is it ok if the configuration cells were hidden from getting displayed on the docs website and reveal when people click view on colab on colab page?

lorenzoh commented 2 years ago

Sorry for letting this fall through the cracks!

Hey, I made a repo that contains the code for converting a normal ipynb notebook to the one that can run on Colab. https://github.com/Chandu-4444/Colab.jl This is a sample one, there are few improvements to be made (Specifying the output file location etc).

Thanks for starting on this

I have an idea to do this. Is it ok if the configuration cells were hidden from getting displayed on the docs website and reveal when people click view on colab on colab page?

They should be added to the notebook, and the documentation system will take care of hiding the setup cells as well as the "Open in Colab" button

Chandu-4444 commented 2 years ago

They should be added to the notebook, and the documentation system will take care of hiding the setup cells as well as the "Open in Colab" button

So, shall I replace all the current notebooks with the notebooks containing the config cells that can run on Colab?

Chandu-4444 commented 2 years ago

Colab.jl is now a registered Julia package. I think this can now be integrated into docs?

lorenzoh commented 2 years ago

Hey, sorry for letting this sit again, integrating it into the docs is bottlenecked on my time making an integration with Pollen.jl right now. In any case, thanks for getting this running! I'll open an issue on the repo to propose some changes when I get around to it

FluxML / FastAI.jl

Colab integration #133