This has been deployed for the past week or so and nothing bad happened (...after I worked out all the initial kinks -- if you saw out-of-space errors followed by me prodding ofborg to re-eval... no you didn't).
We now run 3 evaluators on each of the 5 machines we have access to through EM. I saw about 160GiB peak when all 3 were running the outpaths evaluation at the same time, out of a total of 256GiB of RAM. This (hopefully) leaves room for the single builder that resides on the same machine as well as for the outpaths check to consume even more ( :roll_eyes: ) memory.
This also seems to more than handle the (current) Nixpkgs PR cadence, as all the machines chewed through the ~160 queue in about 10 hours:
We also run git gc every time we clone / before we clone so as to help curb the inode usage of these perpetual Nixpkgs checkouts. Before this, I had seen checkouts of more than 70GiB on disk!!! It's not fast (I think it takes 10 minutes tops), but it certainly helps cut down on inode clutter.
This has been deployed for the past week or so and nothing bad happened (...after I worked out all the initial kinks -- if you saw out-of-space errors followed by me prodding ofborg to re-eval... no you didn't).
We now run 3 evaluators on each of the 5 machines we have access to through EM. I saw about 160GiB peak when all 3 were running the outpaths evaluation at the same time, out of a total of 256GiB of RAM. This (hopefully) leaves room for the single builder that resides on the same machine as well as for the outpaths check to consume even more ( :roll_eyes: ) memory.
This also seems to more than handle the (current) Nixpkgs PR cadence, as all the machines chewed through the ~160 queue in about 10 hours:
We also run
git gcevery time we clone / before we clone so as to help curb the inode usage of these perpetual Nixpkgs checkouts. Before this, I had seen checkouts of more than 70GiB on disk!!! It's not fast (I think it takes 10 minutes tops), but it certainly helps cut down on inode clutter.It's not beautiful, but it works.