search

mediacloud / story-indexer

The core pipeline used to ingest online news stories in the Media Cloud archive.
https://mediacloud.org
Apache License 2.0
2 stars 5 forks source link

Close all write operations to S3 #316

Closed pgulley closed 3 months ago

pgulley commented 4 months ago

We are aiming to be S3 free as soon as possible- the most important step is that we stop writing new data into S3.

On the story indexer, we are currently:

philbudne commented 3 months ago

Regular production stack now writing to just B2. The rss-indexer stack is still draining its queue, and is still writing to S2 and B2.

My understanding from this mornings meeting is that ILM backups are only writing to B2, so I think this task is done.

ivanlawrence commented 3 months ago

I'm an elasticsearch 8.15 user and am hitting a wall with configuring BackBlaze as a repo (see forum posts: https://discuss.elastic.co/t/cannot-register-snapshot-repository-for-s3-compliant-backblaze-service/365581/4) any chance I can get some love as I too want to reduce my backup costs by using BackBlaze <3

philbudne commented 3 months ago

@ivanlawrence I've replied at https://discuss.elastic.co/t/cannot-register-snapshot-repository-for-s3-compliant-backblaze-service/365581/6?u=phil_budne