The copy of the files into the cluster occurs file by file in a multithreading parallel for each batch ssh connection.
This is not efficient for two reasons:
It provokes many simultaneous connections to the cluster, risking a ban of the IP for a DoS attack.
It is super slow when thousands of configurations are required to be copied.
It is quite straightforward to improve, as the input files can be tarred in one file, copied to the cluster, and then extracted there.
Maybe it would be very good if a flag in the cluster configuration can trigger this option when available.
It would speed up the I/O with the cluster when many configurations need to be computed.
mesonepigreco
commented
2 years ago
The copy of the files into the cluster occurs file by file in a multithreading parallel for each batch ssh connection. This is not efficient for two reasons:
It is quite straightforward to improve, as the input files can be tarred in one file, copied to the cluster, and then extracted there. Maybe it would be very good if a flag in the cluster configuration can trigger this option when available. It would speed up the I/O with the cluster when many configurations need to be computed.