search

mar10 / pyftpsync

Synchronize directories using FTP(S), SFTP, or file system access.
https://pyftpsync.readthedocs.io
MIT License
117 stars 25 forks source link

ERROR: [Errno 17] File exists #74

Closed bustekuchen closed 1 year ago

bustekuchen commented 2 years ago

Hi,

Python version: 3.8.10 pyftpsync version: 3.1.0 (also tested with 4.0.0-a3)

Started with the following Error from the Title:

$ pyftpsync download -v --progress . ftps://ftp.someserver.de
Download to /opt/workspace/working
                from ftps://ftp.someserver.de/
Encoding local: utf-8, remote: utf-8
No credentials found in keyring('pyftpsync', 'ftp.someserver.de').
Using credentials from .netrc file: user_for_sync:***.
Login as 'user_for_sync'.
Could not send 'OPTS UTF-8': '502 Unknown OPS format.'
Sent 'OPTS UTF8 ON'.
Setting FTP encoding to utf-8 (was latin-1).
Server time offset: -15.34 seconds.
EQUAL             =  [01_Nara_SP]
SKIP UPLOAD       >  transfer.log.all
EQUAL             =  [test]
SKIP UPLOAD       >  transfer-ftp-old.tar.gz
SKIP UPLOAD       >  transfers_non_nagios.log
COPY UNMODIFIED   <  [..]
ERROR: [Errno 17] File exists: '/opt/workspace'
    <EntryPair([..])>: (missing, unmodified) => copy_remote
Traceback (most recent call last):
  File "/home/engl/.local/bin/pyftpsync", line 8, in <module>
    sys.exit(run())
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/pyftpsync.py", line 242, in run
    s.run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 1268, in run
    res = super(DownloadSynchronizer, self).run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 827, in run
    res = super(BiDirSynchronizer, self).run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 213, in run
    res = self._sync_dir()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 558, in _sync_dir
    res = handler(pair)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 851, in on_copy_remote
    self._copy_recursive(self.remote, self.local, remote_entry)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 333, in _copy_recursive
    dest.mkdir(dir_entry.name)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/targets.py", line 392, in mkdir
    os.mkdir(path)
FileExistsError: [Errno 17] File exists: '/opt/workspace'

Then I thought, well what if just have a Problem with relative paths like .. and . Therefore I started pyftpsync again but this time with this exclude -x ".,..". Now things getting stranger. I do get to the part where file operations happen. pyftpsync is creating the first folder, just to immediately afterwords trowing the same error again. But if you just restart he is going one folder further. So with every time I start the program I get one Folder closer to my remote structure.

$ pyftpsync download -v --progress -x ".,.." . ftps://ftp.someserver.de
Download to /opt/workspace/working
                from ftps://ftp.someserver.de/
Encoding local: utf-8, remote: utf-8
No credentials found in keyring('pyftpsync', 'ftp.someserver.de').
Using credentials from .netrc file: user_for_sync:***.
Login as 'user_for_sync'.
Could not send 'OPTS UTF-8': '502 Unknown OPS format.'
Sent 'OPTS UTF8 ON'.
Setting FTP encoding to utf-8 (was latin-1).
Server time offset: -15.37 seconds.
EQUAL             =  [01_Nara_SP]
SKIP UPLOAD       >  transfer.log.all
EQUAL             =  [test]
SKIP UPLOAD       >  transfer-ftp-old.tar.gz
SKIP UPLOAD       >  transfers_non_nagios.log
SKIP UNMATCHED    -  [..]
SKIP UNMATCHED    -  [.]
COPY UNMODIFIED   <  [02_Wakayama_SZ]
ERROR: [Errno 17] File exists: '/opt/workspace/working'
    <EntryPair([02_Wakayama_SZ])>: (missing, unmodified) => copy_remote
Traceback (most recent call last):
  File "/home/engl/.local/bin/pyftpsync", line 8, in <module>
    sys.exit(run())
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/pyftpsync.py", line 242, in run
    s.run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 1268, in run
    res = super(DownloadSynchronizer, self).run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 827, in run
    res = super(BiDirSynchronizer, self).run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 213, in run
    res = self._sync_dir()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 558, in _sync_dir
    res = handler(pair)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 851, in on_copy_remote
    self._copy_recursive(self.remote, self.local, remote_entry)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 340, in _copy_recursive
    self._copy_recursive(src, dest, entry)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 333, in _copy_recursive
    dest.mkdir(dir_entry.name)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/targets.py", line 392, in mkdir
    os.mkdir(path)
FileExistsError: [Errno 17] File exists: '/opt/workspace/working'

$ pyftpsync download -v --progress -x ".,.." . ftps://ftp.someserver.de
Download to /opt/workspace/working
                from ftps://ftp.someserver.de/
Encoding local: utf-8, remote: utf-8
No credentials found in keyring('pyftpsync', 'ftp.someserver.de').
Using credentials from .netrc file: user_for_sync:***.
Login as 'user_for_sync'.
Could not send 'OPTS UTF-8': '502 Unknown OPS format.'
Sent 'OPTS UTF8 ON'.
Setting FTP encoding to utf-8 (was latin-1).
Server time offset: -15.33 seconds.
EQUAL             =  [01_Nara_SP]
EQUAL             =  [02_Wakayama_SZ]
SKIP UPLOAD       >  transfer.log.all
EQUAL             =  [test]
SKIP UPLOAD       >  transfer-ftp-old.tar.gz
SKIP UPLOAD       >  transfers_non_nagios.log
SKIP UNMATCHED    -  [..]
SKIP UNMATCHED    -  [.]
SKIP UNMATCHED    -  [.]
SKIP UNMATCHED    -  [01_Nara_SP]
COPY UNMODIFIED   <  [01_Nara_SP/02_T-Stop_Tester]
ERROR: [Errno 17] File exists: '/opt/workspace/working/01_Nara_SP'
    <EntryPair([01_Nara_SP/02_T-Stop_Tester])>: (missing, unmodified) => copy_remote
Changing to ftp root folder to remove lock file: /
Traceback (most recent call last):
  File "/home/engl/.local/bin/pyftpsync", line 8, in <module>
    sys.exit(run())
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/pyftpsync.py", line 242, in run
    s.run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 1268, in run
    res = super(DownloadSynchronizer, self).run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 827, in run
    res = super(BiDirSynchronizer, self).run()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 213, in run
    res = self._sync_dir()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 592, in _sync_dir
    self._sync_dir()
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 558, in _sync_dir
    res = handler(pair)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 851, in on_copy_remote
    self._copy_recursive(self.remote, self.local, remote_entry)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 340, in _copy_recursive
    self._copy_recursive(src, dest, entry)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/synchronizers.py", line 333, in _copy_recursive
    dest.mkdir(dir_entry.name)
  File "/home/engl/.local/lib/python3.8/site-packages/ftpsync/targets.py", line 392, in mkdir
    os.mkdir(path)
FileExistsError: [Errno 17] File exists: '/opt/workspace/working/01_Nara_SP'

So I was like, okay If getting an error if the folder already exists. I add a only mkdir if not exists. This looks like this: in ftpsync/targets.py on line 389 I Changed the def mkdir:

    def mkdir(self, dir_name):
        self.check_write(dir_name)
        path = normpath_url(join_url(self.cur_dir, dir_name))
        if not self.dir_exists(path):
            os.mkdir(path)

    def dir_exists(self, path):
        return os.path.isdir(path)

But the result of the change was a Tried to navigate outside root error from def cwd. At this point I am not deep enough into this to fix the Problem.

I Hope I could explain the Problem.

Alex

mar10 commented 2 years ago

I can't reproduce it here. Can you give two minimal zip files with some source and target data structures that reproduce the problem? Also, what FTP server are you using?

bustekuchen commented 2 years ago

Hi, I tested a bit more around. It seems like the problem is only in the folder creation, as soon as the folder Structure is in place the sync if the files runs flawlessly.

I currently tried to download a whole Structure from ftp to an empty local folder. With the dummy source structure I can reproduce the issue. As soon as the folders are in place the download works. When I delete on of the folders locally and restart the Download I get the File exists error again for the deleted folder.

infos about the current testing environment: Client(pyftpsync): Ubuntu 20.04.4 LTS python: 3.8.10 Server: CrushFTP 10.1

Test structure: test_data.zip

mar10 commented 2 years ago

Strange. It seems to work here, using v4.0.0-a3

ftp_local % pyftpsync download -v --progress . ftps://ds415/  agains a NAS FTP server
Download to /Users/martin/test/ftp_local
                from ftps://ds415/
Encoding local: utf-8, remote: utf-8
No credentials found in keyring('pyftpsync', 'ds415').
Using credentials from .netrc file: martin:***.
Login as 'martin'.
Could not send 'OPTS UTF-8': '501 OPTS command is not defined for [UTF-8].'
Sent 'OPTS UTF8 ON'.
Server time offset: -0.73 seconds.
SKIP UNMATCHED    -  .DS_Store
SKIP UNMATCHED    -  [#recycle]
COPY UNMODIFIED   <  [01_sub_folder]
COPY UNMODIFIED   <  [02_sub_Folder]
COPY UNMODIFIED   <  [05_Sub-Folder]
Wrote 17/1 files in 0 directories, skipped: 0. Elap: 3.29 sec.
ftp_local %

(Same result with sftp: and ftps: protocols).

When I delete on of the folders locally and restart the Download I get the File exists error again for the deleted folder.

This should be classified as local removal and not be downloaded again:

SKIP UNMATCHED    -  .DS_Store
EQUAL             =  [01_sub_folder]
EQUAL             =  [02_sub_Folder]
SKIP UNMATCHED    -  [#recycle]
SKIP REMOTE DEL.  >X [05_Sub-Folder]
EQUAL             =  [01_sub_folder/2022.6.23]
...

I suspect that your server sends '..' and '.' with the list command. Maybe we see more if you run with more logging (-vvv)

I also committed a blind fix, maybe you can test?

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.