Try to fix multiple problems with files coming from Pure

surfedushare / sources-middleware

A middleware that transforms data from sources into a standard format

MIT License

0 stars 0 forks source link

Try to fix multiple problems with files coming from Pure #66

Closed fako closed 5 months ago

fako commented 11 months ago

Problem occurs when "proxy files" is enabled for HvA

peterdubbeldsurf commented 8 months ago

TimeBox 4h for research. Demo environment needed for testing.

fako commented 7 months ago

There are a lot of file warnings. How are we going to fix this on the source side?

I get the feeling this is very much related to the write errors. Those seem to be failed Sentry logging attempts. How are we making this more sustainable? What is the frontend doing??

fako commented 7 months ago

We are going to try to solve Pure image problems and expect these errors to go away or dramatically reduce. Renaming ticket to reflect this.

fako commented 7 months ago

This is now deployed to dev for testing. We need to discuss how/when we want to deploy this to demo and beyond @peterdubbeldsurf @mlagerwerf

fako commented 6 months ago

We should test a complete reset

fako commented 6 months ago

We should also test streaming on frontend

fako commented 5 months ago

I'm not sure we can fix these write errors on our side. The connections get disconnected at Pure's side at least sometimes. As is seen in this event: https://surf-eduservices.sentry.io/issues/5209011742/events/85bab88877fe4487ada6435e73d42619/?project=4505125577228288

Here's a screenshot that shows that these kinds of Pure disconnects happen together with dozens of write errors:

More example errors showing that Pure cuts off responses half way: https://surf-eduservices.sentry.io/issues/5313807052/?project=4505125577228288&query=is%3Aunresolved&referrer=issue-stream&statsPeriod=90d&stream_index=1

fako commented 5 months ago

Playback van videos werkt, maar er is geen streaming. Dit zal geimplementeerd moeten worden in de frontend, maar de stream hopt dan via ten minste 4 "servers". De AWS balancer, de Nginx frontend, de UWSGI backend en Pure. Waarschijnlijk heeft Pure ook nog balancers en verschillende middleware. Dit is in ieder geval het geval bij Hanze, die een eigen middleware tussen Pure en de Harvester heeft gezet.

Hanze voorbeeld: https://client.publinova.acc.surf.zooma.cloud/product/de-gebiedscooperatie-als-model-voor-leven-lang-ontwikkelen

@peterdubbeldsurf