search

nteract / bookstore

📚 Notebook storage and publishing workflows for the masses
https://bookstore.readthedocs.io
BSD 3-Clause "New" or "Revised" License
202 stars 23 forks source link

add initial implementation of publish endpoint using aiobotocore #34

Closed mpacer closed 5 years ago

mpacer commented 5 years ago

I can't get over how straightforward asyncio is making this.

I need to add tests still, but I tried to keep the implementation close to what we know works on the archive side.

mpacer commented 5 years ago

So, it seems that minio doesn't support versioning as a design decision as of 2016 — I haven't seen anything disagreeing with that (and have also looked through the minio docs)

See issue here: https://github.com/minio/minio/issues/1309

So I'm going to remove the requirement to get versioning working in tests.

mpacer commented 5 years ago

If you want to replicate the most critical part of my experiments with versioning you can view the two responses in the logs when you run the tests on https://github.com/nteract/bookstore/pull/34/commits/f862b02e64573e686c3cd54f0ec8acf80a0b1e90.

mpacer commented 5 years ago

I had to slow down the test again for local dev… but I think this is at least kinda testing some aspect of the publishing endpoint. Unfortunately, we won't be able to test it with automatic versioning on the bucket (even if we'll have it on actual s3).

I am not sure why the circle ci testing isn't working. I'm going to not be able to touch this for a little while, but here is my latest run of the tests (as of be0fcec):

(dev) ~/jupyter/bookstore published_endpoint$ npm run test

> bookstore@1.0.0 test /Users/mpacer/jupyter/bookstore
> node ci/integration.js

running bookstore integration tests
[I 00:06:50.189 NotebookApp] Archiving notebooks to s3://bookstore/ci-workspace
[I 00:06:50.616 NotebookApp] [jupyter_nbextensions_configurator] enabled 0.4.0
[I 00:06:50.620 NotebookApp] nteract extension loaded from /Users/mpacer/jupyter/nteract/applications/jupyter-extension/nteract_on_jupyter
[I 00:06:50.622 NotebookApp] Serving notebooks from local directory: /Users/mpacer/jupyter/bookstore/ci
[I 00:06:50.622 NotebookApp] The Jupyter Notebook is running at:
[I 00:06:50.622 NotebookApp] http://127.0.0.1:9988/?token=...
[I 00:06:50.622 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Created bucket bookstore
[I 00:06:51.869 NotebookApp] Saving file at /ci-archived-local-writeout.ipynb
[I 00:06:51.967 NotebookApp] Processing storage write of ci-archived-local-writeout.ipynb
[I 00:06:51.973 NotebookApp] Saving file at /ci-published-local-writeout.ipynb
[I 00:06:51.992 NotebookApp] Processing storage write of ci-published-local-writeout.ipynb
[I 00:06:52.008 NotebookApp] Publishing to s3://bookstore/ci-published/ci-published-local-writeout.ipynb
[I 00:06:52.034 NotebookApp] Processing published write of ci-published-local-writeout.ipynb
[I 00:06:52.074 NotebookApp] Done with published write of ci-published-local-writeout.ipynb
[I 00:06:52.076 NotebookApp] Done with storage write of ci-archived-local-writeout.ipynb
[I 00:06:52.077 NotebookApp] Done with storage write of ci-published-local-writeout.ipynb
[I 00:06:52.079 NotebookApp] Saving file at /ci-archived-local-writeout2.ipynb
[I 00:06:52.091 NotebookApp] Processing storage write of ci-archived-local-writeout2.ipynb
[I 00:06:52.094 NotebookApp] Saving file at /ci-archived-local-writeout3.ipynb
[I 00:06:52.108 NotebookApp] Processing storage write of ci-archived-local-writeout3.ipynb
[I 00:06:52.134 NotebookApp] Done with storage write of ci-archived-local-writeout3.ipynb
[I 00:06:52.136 NotebookApp] Done with storage write of ci-archived-local-writeout2.ipynb
[I 00:06:52.206 NotebookApp] Saving file at /ci-archived-local-writeout2.ipynb
[I 00:06:52.216 NotebookApp] Processing storage write of ci-archived-local-writeout2.ipynb
[I 00:06:52.219 NotebookApp] Saving file at /ci-archived-local-writeout3.ipynb
[I 00:06:52.231 NotebookApp] Processing storage write of ci-archived-local-writeout3.ipynb
[I 00:06:52.260 NotebookApp] Done with storage write of ci-archived-local-writeout2.ipynb
[I 00:06:52.261 NotebookApp] Done with storage write of ci-archived-local-writeout3.ipynb
[I 00:06:52.327 NotebookApp] Saving file at /ci-archived-local-writeout2.ipynb
[I 00:06:52.341 NotebookApp] Processing storage write of ci-archived-local-writeout2.ipynb
[I 00:06:52.345 NotebookApp] Saving file at /ci-archived-local-writeout3.ipynb
[I 00:06:52.361 NotebookApp] Processing storage write of ci-archived-local-writeout3.ipynb
[I 00:06:52.391 NotebookApp] Done with storage write of ci-archived-local-writeout2.ipynb
[I 00:06:52.394 NotebookApp] Done with storage write of ci-archived-local-writeout3.ipynb
[I 00:06:52.455 NotebookApp] Saving file at /ci-archived-local-writeout2.ipynb
[I 00:06:52.466 NotebookApp] Processing storage write of ci-archived-local-writeout2.ipynb
[I 00:06:52.470 NotebookApp] Saving file at /ci-archived-local-writeout3.ipynb
[I 00:06:52.483 NotebookApp] Processing storage write of ci-archived-local-writeout3.ipynb
[I 00:06:52.512 NotebookApp] Done with storage write of ci-archived-local-writeout2.ipynb
[I 00:06:52.513 NotebookApp] Done with storage write of ci-archived-local-writeout3.ipynb
[I 00:06:52.590 NotebookApp] Publishing to s3://bookstore/ci-published/ci-published-local-writeout2.ipynb
[I 00:06:52.613 NotebookApp] Processing published write of ci-published-local-writeout2.ipynb
[I 00:06:52.639 NotebookApp] Done with published write of ci-published-local-writeout2.ipynb
[I 00:06:52.642 NotebookApp] Saving file at /ci-published-local-writeout2.ipynb
[I 00:06:52.654 NotebookApp] Processing storage write of ci-published-local-writeout2.ipynb
[I 00:06:52.668 NotebookApp] Publishing to s3://bookstore/ci-published/ci-published-local-writeout3.ipynb
[I 00:06:52.694 NotebookApp] Processing published write of ci-published-local-writeout3.ipynb
[I 00:06:52.724 NotebookApp] Done with storage write of ci-published-local-writeout2.ipynb
[I 00:06:52.726 NotebookApp] Done with published write of ci-published-local-writeout3.ipynb
[I 00:06:52.728 NotebookApp] Saving file at /ci-published-local-writeout3.ipynb
[I 00:06:52.742 NotebookApp] Processing storage write of ci-published-local-writeout3.ipynb
[I 00:06:52.767 NotebookApp] Done with storage write of ci-published-local-writeout3.ipynb
[I 00:06:52.849 NotebookApp] Publishing to s3://bookstore/ci-published/ci-published-local-writeout2.ipynb
[I 00:06:52.872 NotebookApp] Processing published write of ci-published-local-writeout2.ipynb
[I 00:06:52.903 NotebookApp] Done with published write of ci-published-local-writeout2.ipynb
[I 00:06:52.906 NotebookApp] Saving file at /ci-published-local-writeout2.ipynb
[I 00:06:52.916 NotebookApp] Processing storage write of ci-published-local-writeout2.ipynb
[I 00:06:52.931 NotebookApp] Publishing to s3://bookstore/ci-published/ci-published-local-writeout3.ipynb
[I 00:06:52.957 NotebookApp] Processing published write of ci-published-local-writeout3.ipynb
[I 00:06:52.988 NotebookApp] Done with storage write of ci-published-local-writeout2.ipynb
[I 00:06:52.993 NotebookApp] Done with published write of ci-published-local-writeout3.ipynb
[I 00:06:52.996 NotebookApp] Saving file at /ci-published-local-writeout3.ipynb
[I 00:06:53.007 NotebookApp] Processing storage write of ci-published-local-writeout3.ipynb
[I 00:06:53.037 NotebookApp] Done with storage write of ci-published-local-writeout3.ipynb
[I 00:06:53.116 NotebookApp] Publishing to s3://bookstore/ci-published/ci-published-local-writeout2.ipynb
[I 00:06:53.139 NotebookApp] Processing published write of ci-published-local-writeout2.ipynb
[I 00:06:53.171 NotebookApp] Done with published write of ci-published-local-writeout2.ipynb
[I 00:06:53.173 NotebookApp] Saving file at /ci-published-local-writeout2.ipynb
[I 00:06:53.184 NotebookApp] Processing storage write of ci-published-local-writeout2.ipynb
[I 00:06:53.199 NotebookApp] Publishing to s3://bookstore/ci-published/ci-published-local-writeout3.ipynb
[I 00:06:53.226 NotebookApp] Processing published write of ci-published-local-writeout3.ipynb
[I 00:06:53.254 NotebookApp] Done with storage write of ci-published-local-writeout2.ipynb
[I 00:06:53.259 NotebookApp] Done with published write of ci-published-local-writeout3.ipynb
[I 00:06:53.262 NotebookApp] Saving file at /ci-published-local-writeout3.ipynb
[I 00:06:53.276 NotebookApp] Processing storage write of ci-published-local-writeout3.ipynb
[I 00:06:53.306 NotebookApp] Done with storage write of ci-published-local-writeout3.ipynb
[I 00:06:53.383 NotebookApp] Publishing to s3://bookstore/ci-published/ci-published-local-writeout2.ipynb
[I 00:06:53.407 NotebookApp] Processing published write of ci-published-local-writeout2.ipynb
[I 00:06:53.430 NotebookApp] Done with published write of ci-published-local-writeout2.ipynb
[I 00:06:53.434 NotebookApp] Saving file at /ci-published-local-writeout2.ipynb
[I 00:06:53.448 NotebookApp] Processing storage write of ci-published-local-writeout2.ipynb
[I 00:06:53.529 NotebookApp] Publishing to s3://bookstore/ci-published/ci-published-local-writeout3.ipynb
[I 00:06:53.557 NotebookApp] Processing published write of ci-published-local-writeout3.ipynb
[I 00:06:53.584 NotebookApp] Done with published write of ci-published-local-writeout3.ipynb
[I 00:06:53.585 NotebookApp] Done with storage write of ci-published-local-writeout2.ipynb
[I 00:06:53.588 NotebookApp] Saving file at /ci-published-local-writeout3.ipynb
[I 00:06:53.603 NotebookApp] Processing storage write of ci-published-local-writeout3.ipynb
[I 00:06:53.636 NotebookApp] Done with storage write of ci-published-local-writeout3.ipynb
ci-published-local-writeout.ipynb
{"cells": [{"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["import this"]}], "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.0"}}, "nbformat": 4, "nbformat_minor": 2}
Notebook on S3 matches what we sent
[C 00:06:54.418 NotebookApp] received signal 15, stopping
[I 00:06:54.419 NotebookApp] Shutting down 0 kernels
using CI config
ci-archived-local-writeout.ipynb
{"cells": [{"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["import this"]}], "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.0"}}, "nbformat": 4, "nbformat_minor": 2}
Notebook on S3 matches what we sent
ci-archived-local-writeout2.ipynb
{"cells": [], "nbformat": 4, "nbformat_minor": 2, "metadata": {"save": 3}}
Notebook on S3 matches what we sent
📚 Bookstore Integration Complete 📚
jupyter server terminated
mpacer commented 5 years ago

@rgbkrk do you have any thoughts as to why we’d be running into this issue on circle ci when it’s not present locally?

willingc commented 5 years ago

@rgbkrk do you have any thoughts as to why we’d be running into this issue on circle ci when it’s not present locally?

I'm guessing that local and circle ci differ in deployment implementation. I would try making a few of the console.logs more verbose.

rgbkrk commented 5 years ago

Integration tests pass locally for me with this, and as I promised I'll investigate on the Circle CI side.

rgbkrk commented 5 years ago

Commits are going to start coming in as I tackle the Circle CI bits.

rgbkrk commented 5 years ago 
[I 22:23:28.355 NotebookApp] Processing storage write of ci-published-local-writeout.ipynb
cleaning up a rogue jupyter server
unhandledRejection SyntaxError: Unexpected token < in JSON at position 0
    at JSON.parse (<anonymous>)
    at parseJson (/home/circleci/repo/node_modules/rxjs/internal/observable/dom/AjaxObservable.js:364:21)
    at tryCatcher (/home/circleci/repo/node_modules/rxjs/internal/util/tryCatch.js:7:31)
    at parseXhrResponse (/home/circleci/repo/node_modules/rxjs/internal/observable/dom/AjaxObservable.js:370:50)
    at new AjaxErrorImpl (/home/circleci/repo/node_modules/rxjs/internal/observable/dom/AjaxObservable.js:354:21)
    at exports.XMLHttpRequest.xhrLoad [as onload] (/home/circleci/repo/node_modules/rxjs/internal/observable/dom/AjaxObservable.js:309:37)
    at exports.XMLHttpRequest.dispatchEvent (/home/circleci/repo/node_modules/xmlhttprequest/lib/XMLHttpRequest.js:591:25)
    at setState (/home/circleci/repo/node_modules/xmlhttprequest/lib/XMLHttpRequest.js:614:14)
    at IncomingMessage.<anonymous> (/home/circleci/repo/node_modules/xmlhttprequest/lib/XMLHttpRequest.js:447:13)
    at IncomingMessage.emit (events.js:187:15)

This leads me to believe this is an error being returned from the jupyter server. I'll add some logging to the ajax response here.

rgbkrk commented 5 years ago

I think what I might just do here is remove the changes to the integration tests, see about bringing this in as is and then iterating on the test suite to clean it up a bit.

rgbkrk commented 5 years ago

Yeah lets get this in a workable state and try to investigate the error handling with new PRs.