create new tenant in rebuilt stack for bulk load test - Githubissues

samvera / hyku

Hyku: A multi-tenant Hyrax application built on the latest and greatest Samvera community components. Brought to you by the Hydra-in-a-Box project partners and IMLS; maintained by the Hyku Interest Group.

https://samvera.atlassian.net/wiki/spaces/hyku/overview

Other

94 stars 47 forks source link

create new tenant in rebuilt stack for bulk load test #978

Closed hannahfrost closed 7 years ago

hannahfrost commented 7 years ago

Attached zip file contains a csv with druids that can be safely bulk-imported to Hyku for testing. (I confirmed rights clearance with the curator)

StanfordBand-unrestricted-accessioned.csv.zip

mjgiarlo commented 7 years ago

This is currently blocked by the effort to YAMLize our CF templates.

hannahfrost commented 7 years ago

assigned to @eefahy and in progress, right?

mjgiarlo commented 7 years ago

Is this unblocked or is it contingent on the ImageMagick stuff? @eefahy @bbranan

bbranan commented 7 years ago

We've been doing bulk load testing in another stack, in order to allow updates to resolve things like the Image Magick issues. Perhaps this task should just be renamed to say "Run bulk load tests", and we'll continue to do that in the Oregon stack, that way it doesn't impact those using demo for testing.

mjgiarlo commented 7 years ago

@bbranan @eefahy this is in progress, right? Or done? Could one of you assign this issue to yourself and move into the right column on Waffle?

mjgiarlo commented 7 years ago

@bbranan @eefahy @dbernstein :point_up: :question:

bbranan commented 7 years ago

My understanding of the current state:

There has been sufficient testing from DPLA to validate that the ResourceSync implementation in Hyku is functional. No further bulk testing is required here.
The primary issues that are still outstanding from previous bulk loading tests are related to running FITS and ImageMagick on the worker machines due to the amount of system resources required by each. Until changes are made that modify how these processes are executed on the workers, there's not a great deal of value in performing further tests.

If either of these statements is incorrect, please make that known.

mjgiarlo commented 7 years ago

@bbranan OK, sounds good. Thanks!

Do we need tickets for the FITS/ImageMagick-related work?

I'm inclined to close this issue for now.

bbranan commented 7 years ago

I created a ticket for breaking out queues, which will provide a better way to size worker nodes based on the job type: #1209. Once this work is done, it will make sense to conduct further large scale tests.