search

SwissOpenEM / ScopeMArchiver

0 stars 0 forks source link

[Upload] Final test with 10Gb/s network #20

Open phwissmann opened 4 months ago

phwissmann commented 4 months ago

Configuration needs to be like:

VM with IP 129.132.5.229 GW 129.132.5.225 Network Mask 255.255.255.248

You will be in our network and you will be able to transfer data at 10 Gbit/s since you do not cross any firewall.

phwissmann commented 4 months ago

Blocked by not having the VM in the right network

phwissmann commented 3 months ago

Waiting for VM to move to ScopeM network

phwissmann commented 2 months ago
Web based: File size Checksum enabled Rate
10GB y 1 Gbps
10 GB n 3.5 Gbps

Expected (?) max rate: 10 Gbps

Minio go application: File size Rate
10 GB 3.5 Gbps
sbliven commented 2 months ago

Can you test uploading from localhost? That would check if it's limited by minio, disk speed, etc.

Is production expected to be a similar VM or will it have more hardware?

phwissmann commented 2 months ago

It's harder than it probably should be to do on the vm... but on my laptop I get similar or slightly lower speeds. Looking at this https://github.com/dvassallo/s3-benchmark I think it's safe to say that the s3 protocol/go sdk is very capable and the backend is the bottleneck.

The hardware/setup has not yet been optimized. From what I know this can scale in terms of compute as well as add some SDD storage.