Design & Plan to add Local Filesystem cloning

[mpacer]

Proposal - Local Filesystem Cloning

This proposes adding the ability to clone from files beneath a privileged directory on your filesystem to your local directory.

Current Limitation

The Jupyter ContentManager API is not normally allowed to access anything outside of its base directory. This prevents users from accessing files above the base directory.

Use Case

The use case arises in cases where you want to provide access to content that is not normally available to the Jupyter Contents API.

For example, suppose that you were running JupyterHub for a number of users and had a filesystem of content that all users should have access to in a readonly way.

Workflow option - Rejected

You could display the contents with commuter, but users would then need to download and upload the content to their local server to be able to get an executable copy of the content. You could mount the file system beneath their Jupyter directory which would let you copy the contents. But that could lead to confusion and clutter in the user's filesystem.

Proposed Workflow

File system cloning would allow you to mount the volume at a in a standard location not beneath the users' base dir, and provide a way for users to directly clone that content as a copy into their local directory. Commuter could then show a clone button that would trigger this API, massively simplifying the workflow needed to go from a readonly to an executable copy of a notebook.

Key Benefit: This allows a similar workflow to be shared by both content hosted on s3 (what bookstore currently supports) as well as content on a filesystem.

^ Edited: The above description was added based on a super helpful suggestion from @willingc

A proposed model for filesystem cloning:

Design

• filesystem cloning is disabled by default (any default would potentially pose a security issue)
• filesystem cloning will only work for files that are children of a specific base directory (fs_cloning_basedir)
• cloning source will be defined using a relative path from the fs_cloning_basedir
• we will use the notebook contents manager to handle security policies around the cloning target path
• separate validation checks for s3 cloning and fs cloning
• create new /api/fs-clone and fs-clone endpoints with relpath query parameter and model key
• modify clone.html template to be used by both sets of handlers

Plan

• [x] Add traitlet for setting filesystem cloning base directory #155
  • [x] Add validation check for filesystem cloning; change existing cloning validation to be s3 specific #155
  • [x] Add tests for validation checks #155
  • [x] Update Configuration docs #155
  • [x] Expand validation checks to log when server is misconfigured #159 #160
• [x] Create new BookstoreFSCloneHandler with relpath query params #158
  • [x] Update clone.html template to be more generic #156
  • [x] Update REST API docs #161
  • [x] Update API docs #158
  • [x] Create new handler unittests #158
  • [x] add /bookstore/fs-clone endpoint #159
• [x] Create new BookstoreFSCloneAPIHandler with relpath and target_path model params #163
  • [x] Update REST API docs #164
  • [x] Update API docs #163
  • [x] add /api/bookstore/fs-clone route & update handler tests #164
  • [x] Create new handler unittests #163
    • [x] Test for invalid notebook contents #163
    • [x] Test for trying to target outside of basedir #163
    • [x] Test for trying to get source notebook outside of that specified by fs_cloning_basedir #163
    • [x] Create test_files dir with an example EmptyNotebook.ipynb #163
    • [x] Create successful clone test for EmptyNotebook#163
• [x] Add utility method for handling common relpath validation patterns #157
  • Test for
    • [x] escaping out of base directory #157
    • [x] submitting absolute path #160
• [x] Improve S3 cloning integration testing #162
  • [x] Add S3 clone test for the post endpoint #162
  • [x] Add cleanup methods for the notebook files created. #162
• [x] File system cloning integration testing #165
  • [x] Add in tests for fs cloning to integration.js #165
  • [x] Create target example ipynb file in /bookstore/tests/test_files directory #165
  • [x] Update server configuration to target /bookstore/tests directory #165

@chyzzqo2, does this sound like a good plan?

Replaces #110.

[willingc]

@mpacer Thanks for updating the design and plan. I've added a few headings for readability for devs in the future. :smile:

[mpacer]

@willingc Thank you so much for adding the structure — it really does make this much easier to read!

[mpacer]

So happy we got through all this so quickly thanks @willingc and @chyzzqo2 for helping make that happen!!