Poor performance

[mottosso]

At the moment, running any command via avalon takes at least 10 seconds before any output appears. Starting the launcher takes about 30 seconds, and within Maya running Pyblish QML takes a good minute or so.

That's bad.

I suspected it was due to mounting via Samba, but even running it from the native UNC path is terribly slow. So either UNC is in fact as slow, or the Docker volume is slow. Beyond that I'm not sure what it could be.

Alternatives I can think of is bundling Python and its dependencies into less files, as I suspect the problem is not mb/sec but rather files/sec; in which case one large file would be fast, whereas many small files are slow.

[mottosso]

Basic test of merely copying files via Windows Explorer shows the following.

1. 1GB file from A:\ to local directory, 100MB/s
2. A:\bin\windows\python36 -> local directory, 0.2-3.0MB/s at a rate of ~20 files/sec

1GB file can be created via cmd.exe like so.

fsutil file createnew fakefile.txt 1073741824

A:\ is a mounted Samba share, hosted in a Docker container running on a VirtualBox machine at 4 cores, 4GB or RAM and the Intel PRO/1000 MT Desktop adapter.

[mottosso]

Switching Docker host from VirtualBox to VMWare yields similarly poor results of 0.8-5MB/s at ~25 files/sec

[mottosso]

Found a FUSE equivalent for Windows called WinFSP which can be used to write custom file systems, one of which is sshfs, available on all platforms. I mounted it to X:\.

Overall, twice the speed of Samba but still 10x slower than the local file system. On tests, slower than Samba on both large and small files, so not entirely sure how Avalon can launch faster. I suspect it's faster at reading small files, like Python modules.

Launch Avalon

1. From local, 2.7s
2. From x:\, 19.4s
3. From a:\, 44s

Large file

1. 1GB file from local -> local, 170 MB/s
2. 1GB file from a:\ -> local, 100MB/s, where a:\ is Samba
3. 1GB file from x:\ -> local, 45 MB/s, where x:\ is mounted with WinFSP and SSHFS

Many files

1. Many files from local -> local, 18s
2. Many files from a:\ -> local path, 87s
3. Many files from x:\ -> local path, 105s

Test

import shutil
import time

def copy():
     t0 = time.clock()
     shutil.copytree(r"x:\bin\windows\python36", r"c:\avalon\temp\python36")
     t1 = time.clock()
     print("%.2f" % (t0 - t1))

Container

Expose a local directory through ssh via..

docker build -t sshfs .
docker run -ti --rm -v $(pwd)/volume:/volume -p 23:22 --name sshfs sshfs

And then mount \\sshfs\avalon@192.168.99.100!23

FROM ubuntu

RUN apt-get update && apt-get install -y \
    openssh-server \
    openssh-client

RUN mkdir /var/run/sshd && \
    useradd -c 'Avalon user' -m -d /volume -s /bin/bash avalon && \
    echo 'avalon:default' | chpasswd && \
    sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config && \
    # SSH login fix. Otherwise user is kicked off after login
    sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd && \
    echo "export VISIBLE=now" >> /etc/profile

ENV NOTVISIBLE "in users profile"

EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]

Next

Give this a try using Docker for Windows. The performance bottleneck with sshfs may be with VirtualBox.

[mottosso]

Some progress on this.

On the assumption that the bottleneck was Python and it's many modules, I put together an installer-like process for first-time use of Avalon.

Running Avalon for the first time..
Initialising.. 84%..

The process takes about 30-60 seconds and copies Python and dependencies that aren't expected to change very often, such as PyQt5, to the users HOME directory, i.e. ~/.avalon.

It works quite well, and updates to these underlying critical components are expected to happen less than once a year or so, in which case the user can simply erase the .avalon directory following which Avalon would copy the files anew.

However, on doing this the bottleneck wasn't gone! As it happens, at least half of the time spent booting up the Avalon Launcher comes from the fact that Avalon's own libraries, such as Core, is part of the bottleneck. Samba is simply horrendous at dealing many small files over a mounted drive (or even a UNC path for that matter).

The poor performance of launching even just the Python interpreter is of major concern so I won't rest just yet. Next thing I'll try is installing not just critical dependencies, but the Avalon libraries as well. It means requiring an update-mechanism of sorts in place. Something to (1) keep track of which version is currently installed and (2) update when necessary or on-demand.

Launching anything from a locally available Python is near-instantaneous and is how Avalon is intended to work.

[tokejepsen]

Good to hear progress on this.

[mottosso]

The next step I have in mind is replacing the avalon/files container with one that (1) downloads a release from this repo, (2) a release which is effectively the /volume directory gzipped and (3) deploys it into the users home directory.

That should retain the simplicity of Docker installation/use method, whilst reaping the performance of a local install.

For starters, updates could be (1) us making a new release and (2) the end-user deleting the ~/.avalon directory and re-running Avalon. From there we should be quite able to properly test this approach overall, and if all is well can continue having a look at a more automatic updating process.