Store index in S3, write index less times, and implement graceful shutdown

[pietroalbini]

This PR changes how we read and write the index, to improve performance and remove reliance on EFS in production. The changes included in this PR are:

• Read and write from S3 instead of disk when the index path is s3://{bucket}/{path}.
• Implement graceful shutdown and persist the index on shutdown.
• Since we now persist the index at shutdown, persist the index during learning only if the index was not already persisted in the past hour. This should not have an impact during day-to-day use, as graceful shutdown will take care of persisting the latest changes of shutdown, but will serve as a backup if graceful shutdown doesn't work for some reason.

This PR is best reviewed commit-by-commit.