ndushay
commented
6 years ago What are the performance requirements?
-
Queries will usually be by druid
-
Object inventory
- needs to be as fast as possible
- There will be frequent full audits
- There may be continuous audits - as soon as it finishes, it starts over
- … For continuous audit process, it will be going through every object as fast as it can
-
TCR - only called during fixity checks (and to write fixity info)
- Reasonable write speed
- Better read speed to support
- Fixity checks triggered by:
- Audit process
- Ingest process
- About 5G of raw data for 1.5 million objects (See SWAG sizing https://docs.google.com/spreadsheets/d/1myig9TckA-XIr7eYfvQJB0lTcMp37whHaWRqfREysco )
- Audit process does not always do a fixity check - fixity checks only done on a calendar basis or time lag basis (e.g. if 24 hours has passed since last fixity check)
- If object has lots of versions, we get a checksum for each version … so scales linearly with numbers of versions for objects
- might mitigate with only looking at latest version
-
Provenance - huge data store, rarely read
- Can be sluggish
- Maybe elasticsearch or kafka
-
SDR web services (to inform argo??) - can be slow
[ ] which storage options can meet the performance requirements
see https://docs.google.com/document/d/1iiMKlBQmlFlKC-VusQXGjZLjwq39mhU4qnsDOu4t0xE notes under "what is the reasoning behind separating the 3 pieces of the PCC?"
[ ] will we need multiple database instances for different performance needs?