Consider reverting pipelines to using a Docker instance of SDS as we did previously. The SDS service is frequently running out of memory and has problems when it's left standing for long periods. Using EMR wouldn't incur this problem.
A full ingest used to take an hour in production with Docker and now takes 10.
peggynewman
commented
11 months ago
Consider reverting pipelines to using a Docker instance of SDS as we did previously. The SDS service is frequently running out of memory and has problems when it's left standing for long periods. Using EMR wouldn't incur this problem. A full ingest used to take an hour in production with Docker and now takes 10.