search

dulibrarytech / digitaldu-backend

backend app to the DU digital collections repository
Apache License 2.0
1 stars 0 forks source link

archivesspace API errors when making too many requests for batch metadata updates #203

Closed kimpham54 closed 4 years ago

kimpham54 commented 4 years ago

412 code after making many requests to the API - Fernando finished the metadata batch update by collection, but there seems to be potential memory issues or thread issues every time you make API requests. wants to schedule updates where things don’t explode. server is fine but handling requests becomes an issue when there are too many consecutive ones. possible apache issues? not hardware issue but app related (issue was reproducible in local environment). maybe also look into cron jobs, scheduling updates on a monthly basis

freyesdulib commented 4 years ago

complete. Worked around issue by importing 10 records at a time with each batch generating and destroying its own session token. Added timer to allow archivesspace to throttle down CPU before next batch begins. Cronjob is in place to run weekly.