Increase US rendering capacity

[pnorman]

pyrene, the one US rendering server, no longer has the capacity to keep up with its demand. (https://github.com/openstreetmap/operations/issues/625#issuecomment-1146706362, and other reports) I was able to relieve some of the pressure by sending significant east coast US traffic to Europe, but sending IAD, EWR, LGA, YYZ, and MIA across the Atlantic is not ideal.

I see three options

1. Get some donated capacity, ideally on the US east coast. If someone reading this is able to help or can point us at someone to help, please email operations@osmfoundation.org. Anywhere in the US or Canada would work.
2. Buy a new server and send it to OSUOSL
3. Buy a new server and find a location on the East Coast to host

My preferred order is 1, 2, 3. I would rather not add another location we have hardware in for just one server.

[anon8675309]

Would more people seeding the torrents help with this issue?

If so, improving the instructions may help get more adoption. I attempted to set up a seed box with deluge, but it doesn't support RSS out of the box and I have yet to get the plugin to work. I'd be happy to write up instructions and contribute them if I can get it working.

[tomhughes]

No this has nothing to do with torrents.

[akadouri]

Do you have rough guesses for what 1 and 2 would cost?

[pnorman]

1 would have no costs for the OSMF. 2 and 3 cost about 7k USD in the past when we got new servers in Europe, but supply shortages will have increased costs since then.

[maswan]

FYI: Our (ACC, nidhogg) network provider tells us that for them north american capacity is cheap and plentiful, so the only downside for shifting the load to the renderer we host is the increased latency (and risk running out of nidhogg rendering capacity).

[ke4qqq]

AWS would be interested in helping provide capacity. Feel free to email djnalley@amazon.com and cashsame@amazon.com and we'll start a conversation.

[Firefishy]

@ke4qqq Thank you. I will reach out to you via email shortly.

[pnorman]

Current plans are to set up a rendering server on AWS (#682) but to also replace pyrene, which has a CPU that is 9 years old, spinning hard drives, and only 2.8TB in RAID5. We can use this issue to track replacing Pyrene.

Pyrene is owned by OSM US.

• [ ] Confirm with OSUOSL that we can have another machine hosted there
• [ ] Reach out to OSM US to see if they're interested in replacing the machine, or if the OSMF would be getting an additional machine
• [ ] Figure out specs. My inclination is we want >4TB of storage over the lifetime of the machine

[mcawley]

In conversations for locating a server in Arizona and also getting University of Arizona support to price out what we need as far as funding. Will connect via email.

[pnorman]

Current DB size is 1.1TiB, est. size in 5 years is 2TB. The tile store is tougher to measure, because it will consume as much space as its given, and more space is always good for cache hit ratios. We run an daily cleanup job that, if >88% of space is used, removes files not accessed in the last 2 days until 80% of disk space is used.

pyrene, with about 1.3TiB of tile store is having to do automated sweeps of old tiles multiple times per day, too often. Nidhogg and Culebre, splitting the metatiles between them, have about 2TiB of tile store each, and take 2-4 days to get from 80% to 88%.

Given the above, I would want >2TB of tile storage for a US server in 5 years, so a total >4TB, which means a 7.68TB disk.

[pnorman]

piasa is now running, so we have enough capacity, and it should remain enough once pyrene is shut off.