search

ProjectOpenSea / stream-js

A TypeScript SDK to receive pushed updates from OpenSea over websocket.
https://docs.opensea.io/reference/stream-api-overview
MIT License
165 stars 51 forks source link

Missing Events #393

Open pixelpawnshop opened 4 months ago

pixelpawnshop commented 4 months ago

Hello, I did some testing on the stream api and compared streaming all offers () with streaming offers on a specific collection in this case it was boredapeyachtclub. What I have found is that there are missing events even though both streams were up (no disconnection). What is interesting is that the streaming all offers () was missing way more events. I tested this couple of times running for up to 30 minutes. In one run the specific offer stream api (boredapeyachtclub) got 21 events for a item and streaming all offers (*) only got 4.

How do you guys handle these missing events? Again I dont mean missing events due to a disconnect I mean simply missing events while the stream is connected.

Couple questions here:

Are these missing events a local thing? Or are these missing offers a global event? If they are local to a single connection would it make sense to open I dont know 10 websocket connections to make sure no event is missed?

If the missing of events is global how do you handle them?

Any idea / help is appreciated!

pixelpawnshop commented 4 months ago

I did further testing, and I would say it is nearly impossible that (*) is missing significantly more events compared to (boredapeyachtclub) only by chance. Especially on those high activity collections nearly every item I compare (boredapeyachtclub) always has more events than (*). What could be the reason for this? Also, what workarounds do we have? Is it a viable option to open multiple connections as I mentioned earlier, or would all connections miss the events regardless?

ryanio commented 4 months ago

Thanks for the report, we'll look into this. If you can test over a couple days and still see issues please let us know in this thread, and any patterns you may notice. There was some instability earlier today but it should be resolved now.

ryanio commented 4 months ago

You can always use the rest api to query events

nq commented 2 months ago

Same issue here. Stream API open and lots of missing events. any updates? @pixelpawnshop

pixelpawnshop commented 2 months ago

@nq os is aware of the issue but I think fixxing this takes some work. So the only workaround currently is backfilling events using https://docs.opensea.io/reference/list_events_by_collection which kind of sucks as this ofc takes a ton of api call to keep it updated. But thats the only solution as for now. All the bots using the same approach.

github-actions[bot] commented 18 hours ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions. If you believe this was a mistake, please comment.