Year 1 Development Phases

[kabilar]

• Phase 1
  • [x] Prototype deployment with Heroku
  • [x] Prototype deployment on AWS EC2
• Phase 2 - Production
  • [x] Configure CircleCI deployment (i.e. Docker image builds)
  • [x] Configure SSL certificate
  • [x] Configure front-end so that AWS Access Key ID and and Secret Access Key are not required by user
  • [x] Update lincbrain.org front-end to include a copy button for S3 URI of an asset
  • [ ] Streamline authentication workflow
  • [x] Create Webknossos object when a LINC API user is APPROVED
  • [ ] Retroactively update existing Webknossos users
  • [ ] For users who are not logged in, update webknossos.lincbrain.org to provide instructions to log into lincbrain.org.
  • [ ] Add webknossos.lincbrain.org link to the homepage top navigation bar
  • [ ] Add fetch mechanism for cookie in the file browser and homepage
  • [x] Automated pushes of Docker images to DockerHub upon pull request merges with CircleCI
  • [x] Currently this is set up to push the master and dev branches to DockerHub. (Can potentially clean up dev tags in the future if necessary.)
  • [x] Configure backups
  • [x] Determine components to backup - Fossildb, Postgres, /binaryData, /persistent
  • [x] Determine backup size
  • [x] Determine frequency
    • Currently backups are daily and moved to Glacier immediately (source code)
    • If backup sizes grow exponentially we can switch to rolling 7 day backups and after 7 days store weekly backups in Deep Glacier.
  • [x] Determine how to load backup
  • [x] Validate backup restore with fresh setup
  • [x] Document back and restore processes
  • [x] Determine maintenance strategy
  • Updates to the container will be manual during announced downtimes.
  • Data migration with new versions will occur by following the Webknossos SQL scripts.
  • Rollbacks (if necessary) will be performed by standing up new instance and moving over the URL.
  • [ ] Determine and optimize scalability and costs
  • [ ] Test with optimized example dataset that is chunked to 32x32
  • [x] Test with 2-3 concurrent users with high-resolution histology Zarr image (7 resolution levels, ~0.4 micron)
  • [x] Increase RAM and CPU resources
    • [x] ~Deploy on r5.2xlarge~
    • [x] Deploy on r5.4xlarge
  • [x] Increase Java Heap Size
  • [x] Increase EBS volume
  • [x] Configure API to work asynchronously when retrieving Zarr chunks from the AWS S3 bucket
  • [x] Turn off instance using AWS Lambda from 8pm - 5am EST
    • [ ] Fix FossilDB version for reboots
  • [ ] Update LINC Docs
  • [ ] Add instructions and naming conventions from Google Doc
  • [ ] Add hyperlinks under a Developer Guide section in the LINC Docs to the respective design and deployment docs.
  • [x] View multiple segmentations simultaneously
  • Segment IDs allow for viewing multiple segmentations in a single layer but the segmentations cannot overlap. For further details see https://github.com/scalableminds/webknossos/issues/8032.
  • It is not currently possible to view multiple segmentation layers simultaneously. For further details see https://github.com/scalableminds/webknossos/issues/5695.
  • Annotations overlap for retrospective data. Users can visualize this data in Webknossos as dataset layers.
  • Annotations will not overlap for prospective data. Users will annotate a single volume/segmentation layer with multiple segment IDs.
  • [x] Layers disappear at higher resolution levels (4-4-1, 2-2-1, 1-1-1)
  • [x] Disable the Upload Datasets tab
  • [x] Disable the Upload Annotation(s) button
  • [ ] Map lincbrain.org datasets to webknossos datasets
  • [ ] Versioning of annotations
    • Multiple annotators for each dataset.
    • For a given dataset and annotator, multiple versions do not need to be saved.
    • [ ] If the underlying dataset layer changes, will the annotations be considered stale?
    • [ ] Histology images will likely not be updated.
  • [x] Update permissions
  • [x] Every user can read every dataset and every annotation. Users is apart of the Default team and set up with Dataset Manager permissions upon account creation.
• Backlog
  • https://github.com/scalableminds/webknossos/issues/7444
• Future work
  • Automate download of volume annotation layers, conversion of wkw to zarr with wkcuber, and upload to lincbrain.org
  • This issue is now tracked in #16. At this point I don't think that we need to automate this process, since it is not clear when annotations would be ready for upload to lincbrain.org. As long as we have a script, we can seamlessly export, convert, and upload the annotations.
  • Clean up current test datasets and annotations
  • This issue is now tracked in #17
  • As apart of our LINC infrastructure monthly expense reports, we will monitor costs and can potentially switch to an EC2 Reserved Instance to reduce costs.
• Not implemented
  • Match the threading options to the instance type
  • Only selected users can have write permissions on select annotations
  • This is not currently needed as multiple annotators will not be editing the same annotation
  • Users have to be manually added to annotations to be able to edit. Setting up this feature would require separate teams.

[aaronkanzer]

@kabilar merged https://github.com/lincbrain/webknossos/pull/13 and merged into production (still need to update dev -> staging -> prod workflow)

Should finalize handling of use case of: Configure front-end so that AWS Access Key ID and and Secret Access Key are not required by user

[kabilar]

Hi @aaronkanzer, I updated our list above. Phases 3 and/or 4 may get pushed into year 2.

[aaronkanzer]

Hi @kabilar -- a couple updates/progress here for Phase 2:

Automated pushes of Docker images to DockerHub upon pull request merges to the master branch

Right now, we are configured to push each branch, no matter master or not -- I think I'd like to keep it this way actually -- as we need to push containers remotely to best test them in staging. Perhaps we can discuss producing a CHANGELOG for what ends up in production

I'm also planning to follow alongside WebKNOSSOS migrations documentation, as it prescribes any updates we might need to do on our end if we sync upstream with our fork

How will we update Docker container? Will there be downtime?

We will have minimal downtime -- this should be as simple as:

1. Pull down the newer Docker image of the API
2. Stop the pre-existing API image, kill image to fully pause
3. Replace the tag referenced in the docker-compose.yml for the API reference link here
4. Run docker-compose up webknossos to launch new version

As long as the newer version is compatible with pre-existing schemas (e.g. postgres, fossildb), this should work as intended. We will have staging as a safeguard for any issues of course.

Configure backups

Currently working through this -- I've been using scalableminds fossildb-client for commands to properly backup and restore see here for code reference -- still stress-testing/confirming that the workflow can restore properly.

We will know more in terms of memory size, but it seems only a couple KB per annotation in FossilDB.

For Postgres, this is the standard pg_dump scenario (similar to LINC Archive).

For both, my hope is to have a cron job on the EC2 instance that exports the backups to S3 routinely. We can discuss the cadence for the cron job.

I'm going to continue to update https://github.com/lincbrain/webknossos/pull/10 with more playbook-ish docs for data restoration/backup -- seems we are in a good place overall.

Let me know if you have any questions in the meantime.

[aaronkanzer]

@kabilar A couple other fun updates (albeit in Phase 4, but still good in terms of understanding how WebKNOSSOS<>LINC will eventually communicate)

In this commit, I introduced SameSite of .lincbrain.org for our WebKNOSSOS auth-related cookies 🍪

(Assuming you are logged into staging), you can now see GET payloads for datasets and annotations for example -- we can ping these endpoints to reflect whatever need-be in LINC Data Platform

A couple of other useful notes here: • WebKNOSSOS datasets are name-unique, thus if "dataset-zarr-hipct" exists, a user can't come along and create "dataset-zarr-hipct" for a different dataset -- this could get tricky for drafts, etc. but something to think about in terms of naming conventions

• We will still need to determine the workflow for how an end user gets the cookie initially -- right now, when you log into WebKNOSSOS, the expiration of the cookie is 1 year -- we can alter as needed. There is a POST route for /auth/login for WebKNOSSOS, but that logic could be more complex -- can look into further.

Nevertheless, good news here in terms of how we want to guide LINC users towards annotations and datasets in WebKNOSSOS

[kabilar]

Awesome, thanks for breaking down each component. This is very helpful. I have some naive questions that we can discuss today.

[aaronkanzer]

Update lincbrain.org front-end to include a copy button for S3 URI of an asset

Handled via code mostly in https://github.com/lincbrain/linc-archive/pull/175

[aaronkanzer]

@kabilar proper backup scripts and cronjob definitions should be all set 📈 https://github.com/lincbrain/webknossos/pull/10/commits/68768f1953085fd88118f240ef35dbd0690ff415