search

offspot / imager-service

Create Kiwix Hotspot microSD cards online
https://imager.kiwix.org/
GNU General Public License v3.0
11 stars 6 forks source link

How complex / expensive would it be to be able to run multiple jobs simultaneously? #241

Closed Popolechien closed 3 years ago

Popolechien commented 3 years ago

There's been a serious uptick in orders, and a lot of them seem to be aiming at 256Gb configs, which take quite a bit of time to execute. Would the ability to run more than one job at once only rely on the purchase of a second configuration (=twice the current cost, even when idle), or is it less linear?

kelson42 commented 3 years ago

@Popolechien The simpliest solution is indeed to have an other worker like https://www.scaleway.com/fr/dedibox/pro/pro-6-xs/, so twice the throughput for twice the price.

The key point is that we need a quite powerfull system with quite a bit of storage (2x the size of the image) and fast storage. My wish would be to imvest in optimisation (in particular in the storage)... but this is unfortunately not that easy, there is no quick win possible. Maybe @rgaudin you can elaborate a bit about what could be done?

rgaudin commented 3 years ago

I don't think we need fast storage. We need fast-enough storage so that it doesn't slows down our downloads too much (where we spend a good amount of time) but if cheap storage allows us to run two (or 3) workers on a single host, that might be worth a shot.

This server is advertised 500 Mbps but from the logs, I see that most of the time we download around 70-80MiB/s, descending to 50MiB and maxing at 90MiB.

Please note that if we have multiple workers on a single host, those 500mbps are then shared. With that HW, I think we can have 3 workers because our QEMU is caped at 4cpu anyway. I think we need a good 600GB par worker to handle 256GB cards.

They have a local disk option with one-time cost for adding additional storage to the server.

I see several interesting options:

Item Size Cost
Disque interne SATA3 3 To 180,00 € HT
Disque interne SSD 960 Go 199,99 € HT
Disque externe USB 1 To 59,00 € HT

It does say: Pour bénéficier de cette option, le serveur doit avoir le niveau de service Business au minimum. Nous vous facturons des frais d’installation mais il n’y aura pas de montant récurrent par la suite. No idea if that's our case.

Popolechien commented 3 years ago

Thanks a lot gentlemen. We have the basic service option so far, business would be an additional 35€/month. The most common image configurations these days are 256Gb, which take ~4 hours at most to create (based on the last 5-6 orders I've looked at). Considering people occasionally put two orders at the same time, we would exceed our promised 24h delivery with 5 clients (or 6 orders) a day. That would be our red line, but then if we reach that level we can largely afford the service.