INTERMITTENT ERROR: MLSD returned unsupported type: {!r}".format(res_type)

[davidhitchins]

I am not having much luck running pyftpsync from either the command line or in a python script. I ran this today (I've replaced the real source and target folders with placeholders) pyftpsync sync --dry-run --exclude=.* SOURCEFOLDER TARGETFTPFOLDER

It failed the first time. Next time it worked and displayed in the terminal window a list of files it would upload (sync) if not in dry run mode. The next time I ran the code (removing dry run mode) I got the same error as the first time. After that I got the error every time either in dry run or normal mode. But it did seem to work once! pyftpsync error.txt Any ideas how to fix?

[mar10]

Could you run a test against a folder that only has one file and use the -vvv option?

[davidhitchins]

Thanks for quick reply. I have now done this they way you recommend (one file in local folder) as well as with the usual folder with many files. With the -vvv option BOTH work in dry run mode but neither works without dry run mode. See attached terminal output. For your information I am using the cli command in a python script pyftpsync sync --dry-run --exclude=.* SOURCEFOLDER TARGETFTPFOLDER because I ALSO could not get the BiDirSynchronizer to work at all (same cause?) I am using A2 hosting. terminal output.txt

[davidhitchins]

Hey, there's something odd. When I ran the command in dry run mode after it succeeded the first time, failed the second time (not in dry run mode) then returned to dry run mode it then failed with that same error message. But it succeeded the first time! Is some file being modified on the server that it doesn't like? It is as if once it fails for a given folder pair in non-dry run mode, it never works again in either mode.

[davidhitchins]

OH DEAR!! Now it is working again in both modes. But I am running the same command using the same folder pairs. So there is something definitely intermittent. What can it be? I will keep trying to analyse the logs. Any suggestions where to look please?

[mar10]

The log shows

Could not parse ''
...
NotImplementedError: MLSD returned unsupported type: None

in the get_dir() command. It seems that intermittently the server ('Pure-FTPd') returns an empty line in response to an MLSD command. I would think that is not compliant with rfc3659, but I am not sure either.

Are you running pyftpsync directly from the source code? In this case we could add some print output and try to skip empty lines, to see if that helps.

[mar10]

You could also try the --ftp-active option to see if it makes a difference

[davidhitchins]

I don't seem to be experiencing the problem any more for some reason. No idea why as I have changed nothing. I wrote a python script - here it is. (You may wonder why I'm not using the simpler BiDirSynchronizer but I could not get this to work without throwing an error every time.) The following fetches the username, password, ftpserver etc from an XML config file. When you say "Are you running pyftpsync directly from the source code? " I am not sure what you mean as I am not really a proper coder. But if you give me a tip I will work out what you mean.

import os
import time
import subprocess
import xml.etree.ElementTree as ET
from datetime import datetime
from shutil import copyfile
import ftpsync

#Set path to xml configuration file
configfilePath = 'E:\\Dropbox\\Projects\\FC scripts\\FTPSYNC\\newmyconfig.xml'

# XML parsing
tree = ET.parse(configfilePath)
root = tree.getroot()
sleep = root.find('waittime').text
def syncnow():
    for ftp in root.find('ftps'):
        ip = ftp.find('ftpaddress').text.split(':')
        username = ftp.find('username').text
        password = ftp.find('password').text
        for folder in ftp.find('folders'):
            local_folder = folder.get('local')
            remote_folder = folder.get('remote')
            print(folder.get('run'))
            if folder.get('run') == "dry":
                command_line = "pyftpsync sync --dry-run -vvv --exclude=.*" + " " + local_folder + " ftp://" + username + ":" + password + "@" + ip[0] +  remote_folder
            else:
                command_line = "pyftpsync sync -vvv --exclude=.*" + " " + local_folder + " ftp://" + username + ":" + password + "@" + ip[0] +  remote_folder

            os.system(command_line)
            return

if __name__ == "__main__":
    syncnow()

[mar10]

You import the ftpsync module, but don't seem to use it programmatically (instead you are calling's the CLI of an already installed pyftpsync installation). Anyway, if you can modify the source code of the pyftpsync installation, it would be possible to add print statements or test conditions.

Since you say you the problem does not occur anymore, my guess is that there are temporary server-side problems, leading to invalid MLSD responses. As long as it is not reproducible, there is not much we can do. I am closing this issue for now. Feel free to re.open if you see need.

[davidhitchins]

Hello. Looks like it is still happening but only at the very end in the process of QUITing. I think all the files are synced correctly but the error still occurs. Can you suggest how I can modify the source code of my pyftpsync installation to try to track down the problem please? final error.txt

THIS IS INCORRECT. It quits as there is an error when enumerating the files in the image folder. See next post (which is accurate.)

[mar10]

Hi,

you could comment in those print statements:

https://github.com/mar10/pyftpsync/blob/a0c0f8f5f076c0d403b76a531382318d5ce4a11c/ftpsync/ftp_target.py#L448 https://github.com/mar10/pyftpsync/blob/a0c0f8f5f076c0d403b76a531382318d5ce4a11c/ftpsync/ftp_target.py#L729 https://github.com/mar10/pyftpsync/blob/a0c0f8f5f076c0d403b76a531382318d5ce4a11c/ftpsync/ftp_target.py#L763

Also the -vvv option might help.

Best would be to set a breakpoint, and step through the code, of course. (If you want to provide a test URL, I could try this also sometime.)