Watchdog uploads at 20MB/s from the Bergamo 2p rig in 422

[rozmar]

@arielleleon has been working on this for a while. The original watchdog implementation is copying to VAST very slowly. (robocopy) Arielle then implemented the VAST api, but that's also very slow.

Meanwhile, copying with windows to vast or using shutil.copytree nearly maxes out the connection (90MB/s)

[jeromelecoq]

The copy is done here : https://github.com/AllenNeuralDynamics/aind-watchdog-service/blob/4020d4406149e7a06cd77ef4cf04f364f341edce/src/aind_watchdog_service/run_job.py#L122C1-L161C20

[jeromelecoq]

watchdog has a nice abstraction of copying so in principle we should be able to get this faster if we figure out what is holding this up.

[jeromelecoq]

From ChatGPT:

There are a few potential reasons why the file copy process could be slow, even when using robocopy, which is generally optimized for better performance:

• Unbuffered I/O (/j): The /j option, which enables unbuffered I/O, is designed to speed up the copy process in certain situations (e.g., when copying large files). However, in cases with small files or high-latency disk systems, unbuffered I/O can sometimes slow things down because it bypasses the file system cache.
• Network Latency: If you're copying files over a network, the speed will depend on network conditions, such as bandwidth, latency, and congestion. Network disk I/O is often a bottleneck, especially if the destination is a network drive.
• Disk Speed: Both the source and destination disks' read/write speeds can affect the overall performance. If the destination disk is slow (e.g., an HDD rather than an SSD) or is highly fragmented, it could delay the copy process.
• File Sizes and Number of Files: Robocopy is more efficient with large files but can slow down if it's copying many small files due to the overhead of creating each file and directory. If you're copying directories with thousands of small files, the /j switch may not be beneficial, and the overall process could be slowed down by file system limitations.
• Robocopy's Retrying Behavior (/r:5): The /r:5 switch indicates that Robocopy will retry a failed file copy operation 5 times before skipping it. If there are access issues (e.g., due to locks or permissions), Robocopy might slow down as it retries the file operation.
• Multithreading: Robocopy supports multithreaded copying using the /MT option (by default, it uses a single thread). If your source and destination systems can handle parallel file transfers, using the /MT flag with an appropriate number of threads (e.g., /MT:16) could significantly speed up the process, especially for large datasets.
• Zipped Mode (/z): The /z option provides restartable copies, which can be slower compared to non-restartable copies. This is more useful for large files being transferred over unstable connections but can add overhead if the connection is stable.

[rozmar]

I also wonder if this comes from the fact that windows task scheduler runs a binary version of watchdog - maybe it would be faster if we just ran it in python directly?

[jeromelecoq]

It is possible the multi-threading is blocked if the windows task scheduler does not allow multiple threads. You are setting up what is available to the agent in windows.

[jeromelecoq]

You could also call the python directly in the scheduler. That is how we were doing QC for ophys for several years and it was working fine.

[arielleleon]

@jeromelecoq Robocopy can be configured to copy faster; this is an exercise in playing with the available parameters in Robocopy as you showed above.

We are looking into s3 protocol commands to execute transfers quickly. Mike T has done this successfully. There are issues with this on the Bergamo rig which are being looked into with IT.

If we want to add a configuration to determine which protocol is used (SMB vs S3) and which application is used for each protocol (shutil vs Robocopy), that id definitely possible. I will leave that for SIPE to determine how they want to do things.

fwiw; robocopy is the only thing I have run on Bergamo. We have been debugging S3 protocol on Bergamo for a while and it has been going slowly because of rig and human (Stuart and me) availability.

[jeromelecoq]

I think it is an important issues as I understood the bergamo team has been building an alternative upload solution aa a result of this issue and found that just changing the python code seem to give faster uploads. Ideally the Bergamo team can help with debugging this issue.

[LKINSEY]

Hi all, Bergamo teammate here. I talked to Marton a little about this and thought I'd do some investigation into this as well because from what I've read so far it does seem calling robocopy from a subprocess should be more efficient at copying files to a network drive than shutil.copytree (what I put into the PyQt6 gui to make copying all data <10 minutes). I am no expert, but after interrogating google and chatgpt, I am suspicious that using an unbuffered I/O might be contributing to some inefficiencies during robocopying. roughly around 50% of the raw data from experiments is <100 MB, from reading online this might cause unbuffered I/O to be less efficient whereas if all of the data was >100 mb, robocopy /j would be really fast and efficient.

maybe we can modify the execute_windows_command function by first sorting the files by size, then call 2 run_subprocess functions inside execute_windows_command, one subprocess copying large files with /j and the other subprocess copying small files without /j

[jeromelecoq]

Ok. @arielleleon can we look into including some of these feedbacks? It is really sad that Lucas and Marton are building their own GUI for this and it impacts our ability to consolidate.