deploy onto main SMG infrastructure - Githubissues

TheScienceMuseum / heritage-connector

Heritage Connector: Transforming text into data to extract meaning and make connections

https://www.sciencemuseumgroup.org.uk/projects/heritage-connector

MIT License

21 stars 3 forks source link

deploy onto main SMG infrastructure #363

Closed kdutia closed 2 years ago

kdutia commented 3 years ago

[x] add to docker-compose
[x] get working on EC2
[x] have a look at SPARQL_ENDPOINT_PROXIED env variable
[x] get thor working from env variable (CORS-proxied SPARQL endpoint)
[x] stress test APIs
[x] add instance and deployment details to README
[x] update auto documentation for APIs

[x] add heritageconnector.sciencemuseum URL (Jamie)
[ ] ~update demos with new path and move all demos and links to new addresses~ - see below

Getting demos working with SSL restrictions - options

[x] switch GH pages over to http using the new domain
[ ] ~try using a load balancer to provide https endpoints to the APIs. Then we can keep using the github pages URLs for the demos.~

[x] Jamie to fix SPARQL_ENDPOINT_PROXIED env variable in EC2
[x] update thor link on demos page
[x] update deta to work with new vectors path
[x] remove Elasticsearch dependencies for APIs
[x] remove 500 error in deployed vectors version. This was just because the entity being tested was missing.
[x] fix issue with vectors API being down

[x] re-deploy onto HC EC2 with monitoring set up to see why it goes down
[ ] setup redirect from deta to EC2 - can we switch bookmarklet over?

kdutia commented 2 years ago

stress test results:

I ran queries of varying sizes and manually monitored RAM and CPU usage using the command docker stats -a

CPU usage stays low and constant for all services.
vectors has a consistent memory footprint, even when fetching a very large number (1000) of nearest neighbours.
fuseki memory usage can hit 3/4GB (~50% of total machine capacity) when faced with very large queries or a large number of queries. This number should go down over time due to Java's garbage collection.

I've added restart:always to both of these services in docker-compose.yml so they should restart after a short break if either of them goes down.

kdutia commented 2 years ago

current infrastructure costs: https://docs.google.com/spreadsheets/d/1uM39y1_KWpTYmsFJLET96-fpMRL_F5IJdPB75BDS0_c/edit#gid=0