AlienVault-OTX / OTX-Python-SDK

The Python SDK for AlienVault OTX
Other
358 stars 162 forks source link

Throttling During High Volume Ingestion #50

Open baalgar opened 4 years ago

baalgar commented 4 years ago

Hello,

I am currently creating a pool of processes that simultaneously hit the API with multiple IOC's to retrieve indicator details.

The API stops pulling after about 670 MB of IOC data.

I made sure my machine is hardwired to the internet and the network has not gone down during my runs.

I have also made sure my machine doesn't go to sleep or that anything is stopping the script from running.

I am handling failing IOC's using exceptions and they get logged with no issues.

After pulling details for about 3-4k IOC's the API hangs.

Any idea what could be causing this issue?

Thank you, Baalgar