search

mamba-org / gator

Conda environment and package management extension from within Jupyter
Other
259 stars 31 forks source link

Add ability to add conda-store packages #169

Closed peytondmurray closed 1 year ago

peytondmurray commented 2 years ago

Summary

This PR adds the ability to add conda-store packages.

Changes

Not sure about the CI failures - they look like an unrelated build issue.

github-actions[bot] commented 2 years ago

Binder :point_left: Launch a binder notebook on the branch Quansight/gator/add-conda-store-packages

fcollonval commented 2 years ago

Hey @peytondmurray I pushed a PR to try fixing the CI

I wanted to test this feature. and as I finally installed conda-store on my computer. I realize that installing package does not work because the panel keep fetching data and every time it does the list is refreshed and the packages selection is lost:

Log from conda-store:

conda-store-server_1  | 172.18.0.1 - - [02/Dec/2021 09:13:52] "GET /api/v1/package/?page=10&size=100&distinct_on=name&distinct_on=version&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [02/Dec/2021 09:13:53] "GET /api/v1/package/?page=11&size=100&distinct_on=name&distinct_on=version&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [02/Dec/2021 09:13:54] "GET /api/v1/package/?page=12&size=100&distinct_on=name&distinct_on=version&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [02/Dec/2021 09:13:55] "GET /api/v1/package/?page=13&size=100&distinct_on=name&distinct_on=version&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [02/Dec/2021 09:13:56] "GET /api/v1/package/?page=14&size=100&distinct_on=name&distinct_on=version&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [02/Dec/2021 09:13:57] "GET /api/v1/package/?page=15&size=100&distinct_on=name&distinct_on=version&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [02/Dec/2021 09:13:58] "GET /api/v1/package/?page=16&size=100&distinct_on=name&distinct_on=version&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [02/Dec/2021 09:13:59] "GET /api/v1/package/?page=17&size=100&distinct_on=name&distinct_on=version&sort_by=name HTTP/1.1" 200 -

Peek 2021-12-02 10-17

peytondmurray commented 2 years ago

Ha, I think you're right - I think I need to be careful when I fetch new packages, and make sure that currently selected packages remain selected during the fetch.

peytondmurray commented 2 years ago

@fcollonval , I think I've got it working. Sorry for the long delay, I've been busy with other projects. I've fixed the behavior of the selected packages. Here's a demo of it working:

https://user-images.githubusercontent.com/14017872/146477097-d15bb4ee-231d-4094-ae6e-e0247c89b014.mp4

fcollonval commented 2 years ago

Indeed the selection is persisted. So now you can look at the next problem, if you try installing packages, you are overwhelming the server with requests:

conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:44] "GET /api/v1/package/?page=518&size=100&distinct_on=name&distinct_on=version&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:44] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:44] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:44] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:44] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:44] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:44] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:44] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:44] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:44] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/package/?page=519&size=100&distinct_on=name&distinct_on=version&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:45] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/environment/default/web/ HTTP/1.1" 200 -
conda-store-server_1  | 172.18.0.1 - - [20/Dec/2021 08:20:46] "GET /api/v1/build/1/packages/?page=1&size=100&sort_by=name HTTP/1.1" 200 -

Using the while loop with no timeout nor throttling of the request should probably be avoided. But moreover you need a way to wait for the build to finish. I don't know the REST API of conda-store. But for this server extension, for long running task, the server endpoint returns 202 code with the location endpoint to request for the task completion status. That status endpoint is then requested periodically (once per second what is already high compare to the time to carry a conda env modification).

https://github.com/mamba-org/gator/blob/master/packages/common/src/services.ts#L866

peytondmurray commented 1 year ago

Closing this PR as stale.