Open pocesar opened 3 years ago
There are definitely plenty of ways we could do this. I would probably start with trying to mimic the queue we have with most of its functionality. Then it could just work the same in crawlers, just instead of calling Apify API, it would call the server. The server could run on the same actor or spawn another actor.
actually, the "Unlimited RQ" is misnomer. it works as a remote dynamic code executor, but faster than RQ / enqueueing, as they are dealt as they arrive. the detail extraction worker code is passed to the server on register.
Brainstorm time. Express server that has 2 endpoints:
/register
eval
s the payload and keep in memory. this is only for not sending the whole function every time on /request. Using the id received here calls the same evaled function using /request/request