rust-lang / crater

Run experiments across parts of the Rust ecosystem!
https://crater.rust-lang.org
627 stars 88 forks source link

Back largest archive by temporary files #716

Closed Mark-Simulacrum closed 8 months ago

Mark-Simulacrum commented 8 months ago

This hopefully resolves the OOMs we see in production each time we try to write a report by keeping the archive on-disk rather than in-memory while we write it. Mid-to-long term I'd like to avoid the mmap and temporary file and instead stream the writing to S3, but this is what I had time to throw together today. I think it should work, though I'm a bit uncertain about the disk backing for temporary files.

Mark-Simulacrum commented 8 months ago

@bors r+

bors commented 8 months ago

:pushpin: Commit 8c79c2f982b035252e558bb7e6493da073d0cc18 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

bors commented 8 months ago

:hourglass: Testing commit 8c79c2f982b035252e558bb7e6493da073d0cc18 with merge 527f48414609d53c97144881d7412262894dc792...

bors commented 8 months ago

:broken_heart: Test failed - checks-actions

Mark-Simulacrum commented 8 months ago

@bors r+

bors commented 8 months ago

:pushpin: Commit ab5c3d70413eff7c37d5e6c0036296f9560f5719 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

bors commented 8 months ago

:hourglass: Testing commit ab5c3d70413eff7c37d5e6c0036296f9560f5719 with merge d3895e4f98e219d60759efd4d96a6df9b3a3d772...

bors commented 8 months ago

:broken_heart: Test failed - checks-actions

Mark-Simulacrum commented 8 months ago

@bors r+

bors commented 8 months ago

:pushpin: Commit a661e16334976777549abe15485570876f83bc9e has been approved by Mark-Simulacrum

It is now in the queue for this repository.

bors commented 8 months ago

:hourglass: Testing commit a661e16334976777549abe15485570876f83bc9e with merge 56535fb6a58e8f302912c75f8b13c0370b115caa...

bors commented 8 months ago

:broken_heart: Test failed - checks-actions

Mark-Simulacrum commented 8 months ago

@bors r+

bors commented 8 months ago

:pushpin: Commit ff335182dea55e3d0ccc9094b2c905d0b8a5c793 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

bors commented 8 months ago

:hourglass: Testing commit ff335182dea55e3d0ccc9094b2c905d0b8a5c793 with merge 76cd44d6dc2ebb4fa5cb66e84031b3e664212660...

bors commented 8 months ago

:sunny: Test successful - checks-actions Approved by: Mark-Simulacrum Pushing 76cd44d6dc2ebb4fa5cb66e84031b3e664212660 to master...