Plan for organizing this repo

[jbednar]

Draft plan for reorganizing this repository for delivery, to be updated as we go along:

• [x] Jean-Luc/Jim: Add instructions for reproducing notebooks, to the README

• [x] Jean-Luc/Jim: Add link to deployment in the Deliverables section of the README

• Improve underlying code (where feasible in time available)

  • [x] PeterW/MattyG/JoeM: Validate TLE computations; what to do with duplicate records?
  • https://github.com/att-vault/vault/pull/14 - Updates to reflect sequence number ordering and 1 min bounds
  • [x] Jean-Luc: commit minor changes already on local disk
  • [x] JimB: add dynspread where appropriate in notebooks
  • [x] JimB: proofread/minor polish HIT_* and Viewing_Tracks
  • [x] Jean-Luc: commit Viewing_AIS_Gaps notebook; if easy, consider suppressing gaps due to edge of data? (in space and time)
  • [x] Jean-Luc: easy improvement: show vessel most recent/soonest future for a selected time (plus optional interpolated position) for Viewing_AIS or Hit_Finder
  • [x] Jean-Luc: Add plots showing colorizing by speed-over-ground, etc.
  • [x] Jean-Luc: Make the hit-finder app fancier with a template that has logos, etc.
  • [x] Jean-Luc: make satellite name dropdown selectable via non-case-sensitive matching, and/or use two-level selection as in UCS_Satellite_metadata.ipynb.
  • [x] Better AIS track selection (often seems like it's broken, but it's just that the click didn't meet the conditions for triggering a selection)
  • [x] Jean-Luc: Make sure that the table display for clicked vessels behaves nicely; expanding one column to read entire vessel name makes it tiny
  • [x] JimB: Release datashader? (if possible this week, but not priority)
  • [ ] Jean-Luc: Datashade all precomputed tracks? Low priority.
  • [ ] Jean-Luc: Make vessel glyphs point in the heading direction? Low priority.

• [x] Categorize, document, and promote three types of notebooks/scripts/apps:

  • EDA (Viewing_*): Start with raw data, reveal it as it is (with as little cleanup as possible, so the same process can be applied to new data)
  • Data-prep (AIS, TLE): Start with raw data, create cleaned/consolidated/computed data
  • Production (Hit_*; really prototypes of production): Start with processed data, do an end-user task (e.g. hit detection).

• Rationalize (clean up and standardize) how files are accessed in the nbs:

  • Which files?
  • [x] For EDA nbs (Viewing*), read raw files wherever practical and normalize column names or content only as much as needed, so that government data scientists can see all the steps required to apply these approaches to their own data.
  • [x] For Data-prep nbs, always read raw files (by design). Should have clear provenance chain from original files.
  • [x] For "production" (well, prototype) apps, preferably read fully processed and cleaned files to show the final workflow as closely as possible. Should use an efficient file format (Parquet, h5, etc.)
  • Where should the files be located (local disk, public S3, private S3)?. How should they be accessed? Plan:
  • [x] Make all notebooks read from ./data/, so that users don't need to edit the notebooks to run them (#15)
  • [x] Master copy of all data files should be in vault-data-corpus on S3, which is public readable.
    • [x] Peter: please put any valuable intermediate files or computed data into vault-data-corpus.
  • [x] On EC2, copy vault-data-corpus to /data, a local directory on the instance
  • [x] On AE5, copy vault-data-corpus to /data, mounted via NFS
  • [x] In github and in anaconda-project .tar.gz files, include a symlink from ./data to /data
  • [x] For other hosts/servers, provide a script to fetch data from S3 and put it in /data (if exists and writable) or else directly in ./data
  • When the above steps are done, a git checkout and anaconda-project should work as-is on Linux and Mac systems with no further step as long as the data exists in /data, while the same process also works even if people can't write to /data
  • [x] The readme should tell people to put data in /data, or if they don't have permissions, ./data.
  • [x] Provide updated Docker images and put into vault-data-corpus
  • [x] Check that data in vault-data-corpus is actually used by our code

• Clean up how notebooks refer to external Python code. Usually this is handled by making a module, but most code here is either in notebooks and not shared with other notebooks, or in independent scripts, and making a module complicates the installation process for private code (can't just put it up on a conda repo), so the plan is:

  • [x] Move vault/Notebook contents directly into vault/, so that immediately a visitor can see that this project is essentially a collection of notebooks (#16, #17)

• Make apps deployable and shareable (reasonable startup times, able to handle multiple users)

  • [x] JimB: Look at pn.state for caching data structures in memory
  • [x] JimB: Add user instructions in each panel app itself, including things to be careful about, if sending URLs to anyone for evaluation
  • [ ] JimB: Optimize user experience and add progress reporting/spinners, to make users appreciate that apps are working even when a step is slow
  • [x] JimB: Make it clear if needed for each app whether it's a prototype full app or whether it's just a showcase of how easy it is to make one-off apps

• [x] JimB: Remove unused notebooks (still will be in git history)

• Deploy apps

  • [x] Plan is to deploy on AE5, but can fall back to EC2 if needed
  • [x] Jean-Luc?: Provide a single index page (ideally with simple screenshots) to link to runnable notebooks and apps
  • [x] Put index URL into README.

• [x] Specify environment

  • [x] environment.yml
  • [x] anaconda-project.yml (can adapt from environment.yml then leave environment.yml as a symlink to this)

• Documentation

  • [x] PeterW: README: steps for reproducing, pointers to all other files
  • [x] JimB: produce PDFs for all proofread notebooks
  • [x] Other doc files

• Deliverables

  • [x] Tagged github repo
  • [x] Zip file is then downloadable as https://github.com/holoviz/datashader/archive/.zip . Result should be a valid anaconda project file deployable in AE5 (whether we mention that or not)
  • [x] Docker image: (Comes free with AE5? Otherwise already demonstrated separately for a Panel app (ask JimB/Raj))

[jbednar]

Closing as that's all that will be done! Docker images are available in S3, and the instructions for them are below.

AT&T Vault Docker images

This directory contains Docker images that can be used to reproduce the 1/2021 AT&T VAULT technical scenario solution and associated analysis files.

Using these images

To run the VAULT notebooks on a local or cloud machine with Docker installed:

1. Download the data locally. Here we'll assume the data is in a directory /vault-data on the host where Docker is running.
2. Download the vault-notebooks.tar.gz image from this directory.
3. Run docker load -i vault-notebooks.tar.gz
4. Run docker run --mount type=bind,source=/vault-data,target=/data -p 8087:8086 vault-notebooks
5. Visit localhost:8087 in a web browser. For security, you'll need to paste in the token that is printed to stdout in step 4.
6. You should now be in a Jupyter session for the default "Viewing_AIS" notebook, where you can do "Kernel/Run All" to run it.
7. To run other notebooks, use "File/Open" in the menu and select one of the other notebooks. "Viewing" and "Hit" should all be runnable as-is using the vault-data-minimal directory; the data-prep and other notebooks may need additional data files and in some cases additional Python packages.

The above steps are sufficient for running the code as submitted for this challenge, dated 7 January and tagged 0.9.0 in github.

Following the demo on 11 January, the setup for the deployed dashboard versions of this code (available already running via bit.ly/attvault) has also been captured, in case the dashboards need to be re-deployed. To run the VAULT dashboards:

1. Download the data locally, if not already, into /vault-data on the host where Docker is running.
2. Download the vault-dashboards.tar.gz image from this directory.
3. Run docker load -i vault-dashboards.tar.gz
4. Run docker run --mount type=bind,source=/vault-data,target=/data -p 8088:8086 vault-dashboards
5. Visit localhost:8088 in a web browser
6. You can now select one of the available dashboards, which should open and run in that browser tab.

Re-creating these images

In case you need to re-create these images from the github source, the steps are:

$ cd /tmp
$ git clone git@github.com:Anaconda-Platform/s2i-anaconda-project.git
$ cd s2i-anaconda-project/
$ make

$ cd ~
$ git clone git@github.com:att-vault/vault.git
$ cd vault
$ s2i build . anaconda-project-ubi7 vault-notebooks -e CMD=Viewing_AIS
$ docker run --mount type=bind,source=/vault-data,target=/data -p 8087:8086 vault-notebooks
$ docker save vault-notebooks > ../vault-notebooks.tar.gz

$ git checkout dev
$ s2i build . anaconda-project-ubi7 vault-dashboards -e CMD=dashboards
$ docker run --mount type=bind,source=/vault-data,target=/data -p 8088:8086 vault-dashboards
$ docker save vault-dashboards > ../vault-dashboards.tar.gz