search

dpovshed / octopus

Sitemap checker/stress test tool based on ReactPHP
11 stars 1 forks source link

Use mq react to manage Queue to load URLs #25

Closed holtkamp closed 6 years ago

holtkamp commented 6 years ago

As suggested in https://github.com/dpovshed/octopus/issues/17

This abstracts a big amount of complexity "away" in an existing component.

holtkamp commented 6 years ago

I noticed (initial) memory consumption increases significantly. https://github.com/clue/reactphp-mq/issues/13

I will keep an eye on this, there is no hurry in merging. Maybe it is better to also keep the "old" approach alive and add the MQ-React Queue as addition... Let's see!

dpovshed commented 6 years ago

Alternatively, it can be a new major release 1.0 .

Since the ability to set SpawnDelay is going to removed, some crazy things may happen - like, all 20-50-70 initial requests will try to get info from server at the exact same time. We do not using spawnDelayMax / spawnDelayMin at the moment, but if needed we can add support quite easily. With a new model - I am not sure how it would work :)

holtkamp commented 6 years ago

Yeah indeed, think it is better to re-think this. Closing for now, will restart my work and try to be a bit cleaner.

holtkamp commented 6 years ago

Since the ability to set SpawnDelay is going to removed, some crazy things may happen - like, all 20-50-70 initial requests will try to get info from server at the exact same time.

The amount of initial request would currently be "capped" at the indicated concurrency setting. I use quite low concurrency settings, but indeed, in case a concurrency of 70 is used, this might result in undesired behaviour.

Asked whether this is possible in https://github.com/clue/reactphp-mq/issues/14