Previously we were storing all project data as we retrieve it from the API, and then wrote to disk at the end. For user's with a large amount of projects, this can exhaust memory. This change writes to disk immediately, before moving onto the next project, without storing the project data in memory over the long term.
Additionally, we cut down parallelization to three (down from seven) to limit the activities happening at any given time.
Previously we were storing all project data as we retrieve it from the API, and then wrote to disk at the end. For user's with a large amount of projects, this can exhaust memory. This change writes to disk immediately, before moving onto the next project, without storing the project data in memory over the long term.
Additionally, we cut down parallelization to three (down from seven) to limit the activities happening at any given time.