This PR contains many changes to optimize the download and processing of event logs. In particular, requests for CSVs containing event logs are nowstreamed and processed using a generator iterator. In addition, when loading the data into New Relic, logs/events are sent in chunks and gc.collect() is called after each chunk which will attempt to free up memory. gc.collect() is also run after flushing the cache.
This PR contains many changes to optimize the download and processing of event logs. In particular, requests for CSVs containing event logs are nowstreamed and processed using a generator iterator. In addition, when loading the data into New Relic, logs/events are sent in chunks and gc.collect() is called after each chunk which will attempt to free up memory.
gc.collect()is also run after flushing the cache.