search

pulibrary / figgy

Valkyrie-based digital repository backend.
Other
36 stars 4 forks source link

readme

Figgy

A digital repository application in use at Princeton University Library for storing and managing digital representations of manuscripts, ephemera, vector, and raster data for export into a variety of front-end displays.

CircleCI Browserstack

Project Setup for Development and Test environments

One-time setup

Follow these steps the first time you clone this project to run in dev or test.

Install Language Dependencies

Install Package Dependencies

Remember you'll need to run bundle install and yarn install on an ongoing basis as dependencies are updated.

Package Setup for Mac M Series Processors

Mapnik currently isn't supported by M-series processors, so yarn install above will fail. To get this working, do the following:

  1. $ arch -x86_64 /bin/zsh --login
  2. you can validate that it's running the right architecture now by viewing the output of the arch command
  3. asdf uninstall nodejs
  4. asdf uninstall yarn
  5. rm ~/.asdf/shims/yarn
  6. asdf install nodejs
  7. npm install -g yarn
  8. yarn install
  9. open a new Terminal or otherwise go back to the arm64 arch.
  10. Add the following to ~/.zshrc or ~/.zshrc.local: 
    # Fix issue with homebrew postgres and rails applications (Figgy in
particular).
# See: https://github.com/ged/ruby-pg/issues/538
export PGGSSENCMODE="disable"

Install Lando

Lando will automatically set up docker images for Solr and Postgres which match the versions we use in Production. The ports will not collide with any other projects you're using Solr/Postgres for, and you can easily clean up with lando destroy or turn off all services with lando poweroff.

  1. Install Lando DMG from [[https://github.com/lando/lando/releases]]

Every time setup

Follow these steps every time you start new work in this project in dev or test

  1. Run bundle exec rake servers:start to start lando services and set up database state.

Running tests

Feature Tests

If you want to watch feature tests run for debugging purposes, you can go to http://localhost:7900, use the password secret, and run tests like this:

RUN_IN_BROWSER=true bundle exec rspec spec/features

Parallel Tests

If you'd like to run the test suite in parallel do the following:

  1. bundle exec rake servers:start
  2. PARALLEL_TEST_FIRST_IS_1=true RAILS_ENV=test rake parallel:setup (Sets up suport database; only needed after db has been destroyed)
  3. ./bin/parallel_rspec_coverage

The output from the parallel runs will be interspersed, and the failures will be listed separately for each parallel run, but final run time and coverage will be reported accurate, and the file that powers the --only-failures flag will be correctly generated.

Development Environment

Load sample development data
  1. Log in to your development instance using your princeton credentials; this creates your user in figgy's db. If you only have user access and need admin access, run bundle exec rake figgy:set_admin_user
  2. Start sidekiq (see below)
  3. rails db:seed # pipe through grep -v WARN to ignore log warnings about the rabbitmq port
Background workers

Some tasks are performed by background workers. To run a Sidekiq background worker process to execute background jobs that are queued:

bundle exec sidekiq
Loading controlled vocabularies

To load the controlled vocabularies in config/vocab/:

Uploading files

By default, Figgy provides users with the ability to upload binaries from the local file system environment using the directory https://github.com/pulibrary/figgy/tree/master/staged_files. One may copy files into this directory for aiding in development, and may upload these files in this directory using the "File Manager" interface (exposed after saving a Work).

Preservation Configuration in Development

Figgy uses Google Cloud Storage buckets for providing support for preserving certain resources. Please find further documentation outlining the configuration for Google Cloud service authentication and permissions management here.

By default, in development, preserved objects will be stored in the directory "tmp/cloud_backup." If you'd like to configure and test Google Cloud storage instead, do the following:

  1. Download, ansible-vault decrypt, and save gcs_pulibrary-staging-credentials.json from https://github.com/pulibrary/princeton_ansible/blob/main/roles/figgy/files/staging-google_cloud_credentials.json (rename to gcs_pulibrary-staging-credentials.json)
  2. Create a .env file in the root with the following settings: 
    STORAGE_PROJECT=pulibrary-figgy-storage-1
STORAGE_CREDENTIALS=tmp/gcs_pulibrary-staging-credentials.json
  3. Restart the server. Now items marked with the cloud preservation policy will save to a bucket you can view at https://console.cloud.google.com/storage/browser
  4. Items only last in this bucket for 2 days, and aren't versioned.

Production tasks

Cloud Fixity Checking

Documentation on setup for staging/production Fixity configuration can be found in preservation_documentation.md.

Cloud Fixity Deployment Steps

  1. gcloud components install beta
  2. gcloud auth login
  3. gcloud config set project pulibrary-figgy-storage-1
  4. cap [staging/production] deploy:google_cloud_function

ArchivesSpace Synchronization and TiTiler functionality

Figgy will persist DAOs to ArchivesSpace on completion of finding aid resources. It also uses an s3 bucket to store geo derivatives and serve them via titiler

To set these up in development, do the following:

  1. lpass login <email>
  2. bundle exec rake figgy:setup_keys

Read-only Maintenance Windows

There are two types of read-only mode.

Read-only Mode

Read-only mode disables writing to the Postgres database. Use princeton_ansible to activate it:

Known issue: In read-only mode users cannot download pdfs (unless they've been cached). See #2866

Index Read-Only

This disables writing to the Solr index, but allows writes to the Postgres database which don't get indexed, such as CDL charges or new user creation. This is most useful for long reindexing operations where we want to minimally impact our patrons.

To enable:

  1. Create a PR which configures index_read_only in config/config.yml for production or staging and deploy the branch.
  2. Deploy main again when reindexing is complete.

Maintaining CircleCI base image

We maintain a Figgy Docker image for use in CircleCI. The Dockerfile is located in the .circleci directory. To update a package, dependency, or ruby version, make edits to the Dockerfile. Then build and push the image to Docker Hub using the following steps (be sure to increment the version):

cd .circleci/
docker login # login to docker hub
docker buildx build --push --platform linux/arm64,linux/amd64 -t pulibrary/ci-figgy:{version} . -f ./.circleci/Dockerfile
docker push pulibrary/ci-figgy:{version}

More

Valkyrie Documentation:

User documentation is maintained in Google Drive:

Links to dependencies used in Figgy:

Other dependencies: