-n -p switches does not seem to work on uploads

[doglover3920]

The reason I wanted to use yafc is the possibility to use -n (newer) and -p (preserve timestamps and permissions).

This works for downloads. So when I download a series of files, only the newer files are downloaded.

However, during an upload these switches seem not to work. The files are always uploaded. And they get the timestamp of the time of upload.

It seem that both switches are not working.

So the upload is still taking place all the time.

Although when I upload to the sftp server with the sftp program the timestamps are kept. But the file still being uploaded, due to the missing -n switch in the sftp program.

Uploading to this server with yafc, (with -n -p) will overwrite all the files, and timestamps are not kept.

[doglover3920]

The long versions seem to be OK:

put --force --newer --resume --preserve *.gz

Willy

[sebastinas]

Uh, yeah, there are some problems regarding argument handling for put. Is it better now?

[doglover3920]

Do not know. I have to get a compiled version for mipsel first. (and I cannot compile it myself)

Willy

[doglover3920]

It still says:

root@et9x00:~# yafc
yafc 1.3.5
This program comes with ABSOLUTELY NO WARRANTY; for details type 'warranty'.
This is free software; type 'copyright' for details.

yafc>

And they are not skipping anything anymore. Yafc just uploads everything

put -f -n *.gz

and

put --force --newer *.gz

Have the same result. Upload everything. And i am certain I only changed only 1 file of the 39 to be uploaded.

Transferred 39 files, total size 1996157318 6iB.

Willy

[doglover3920]

And morerover. the get statement which was OK.

get -f -n -p *.gz

Worked just OK. Now in this version. It downloads also everything.

It seems you broken this.
The get statement was OK. The put was faulty. Now both are faulty.

Willy

[sebastinas]

I'd need a verbose log since I cannot reproduce it.

[TheJosh]

Doesn't --force cause everything to be uploaded?

Also, I use -n daily without issues. On 25/04/2015 2:04 AM, "doglover3920" notifications@github.com wrote:

It still says:

root@et9x00:~# yafc

yafc 1.3.5

This program comes with ABSOLUTELY NO WARRANTY; for details type 'warranty'.

This is free software; type 'copyright' for details.

yafc>

And they are not skipping anything anymore. Yafc just uploads everything

put -f -n *.gz

and

put --force --newer *.gz

Have the same result. Upload everything. And i am certain I only changed only 1 file of the 39 to be uploaded.

Transferred 39 files, total size 1996157318 6iB.

Willy

Reply to this email directly or view it on GitHub https://github.com/sebastinas/yafc/issues/53#issuecomment-95986590.

[doglover3920]

I will try, put when I use: put --force --newer --logfile=/hdd/epg/log.1 .gz or put --verbose --force --newer --logfile=/hdd/epg/log.1 .gz No log file is written.

However managed to create a log with the startup options: yafc --trace=/hdd/epg/log.2 --verbose It is at: http://home.scarlet.be/epgalfasite/log.2 (Too long to be pasted here) All files have been uploaded, although the remote folder contained more recent versions.

BTW is there no option to compare fileSize and make a decision on this argument to update?

Doesn't --force cause everything to be uploaded? I have to use force, because I am using this in batch mode. Unattended. I cannot have confirmations.

[doglover3920]

Got the switches now working for the FTP servers. I think everything hinged on the misconception of the -f (force) option.

However, I have one sftp server which does not behave.

yafc: 'put -n rytec.channels.gz' ** start parsing directory listing -rw-r--r-- 1 1000 1004 59722 Apr 26 07:02 rytec.channels.xml-sat.gz -rwxr--r-- 1 1000 1002 2232 Apr 26 06:54 rytec.channels-greece-en.xml.gz drwxr-xr-x 2 1000 1004 4096 Apr 26 07:12 . -rw-r--r-- 1 1000 1004 137025 Apr 17 00:20
-rwxr--r-- 1 1000 1002 7248 Apr 26 06:54 rytec.channels-denmark.xml.gz -rwxr--r-- 1 1000 1002 8933 Apr 26 06:54 rytec.channels-norway.xml.gz -rw-r--r-- 1 1000 1002 9400 Apr 26 06:54 rytec.channels-sweden.xml.gz -rwxr-xr-x 1 0 0 41 Apr 10 12:07 index.php -rw-r--r-- 1 1000 1004 107324 Apr 26 07:02 rytec.channels.xml.gz -rwxr-xr-x 1 1000 1004 63858 Apr 26 06:54 rytec.channels-sat.xml.gz drwxr-xr-x 3 0 0 4096 Apr 10 11:41 .. -rw-r--r-- 1 1000 1004 10172 Apr 26 06:54 rytec.channels-slovenia.xml.gz -rwxr--r-- 1 1000 1002 10863 Apr 26 06:54 rytec.channels-finland.xml.gz -rwxr-xr-x 1 1000 1004 1354 Apr 26 06:54 rytec.channels-caledonia.xml.gz -rwxr--r-- 1 1000 1002 47497 Apr 26 07:02 rytec.channels.xml-cable.gz \ end parsing directory listing *** added directory '/epg_data' to cache marked directory '/epg_data' for flush error flushing directory 'ˆF' (not cached)

And all these files are uploaded. But the ones uploaded are older than these on the server, They should be skipped. Am I missing something?

Willy

[doglover3920]

Something did not copy properly:

yafc: 'put -n rytec.channels.gz' should be:

yafc: 'put -n rytec.channels*.gz'

[doglover3920]

Using the older yafc I get this:

yafc: 'open kingserver' Looking up rytecepg.kingserver.eu... yafc: 'put -n rytec.channels.gz' rytec.channels.gz: no matches found yafc: 'ls' * start parsing directory listing * -rwxr-xr-x 1 1000 1004 26297 Apr 26 07:16 rytec.channels-cable.xml.gz -rw-r--r-- 1 1000 1004 59722 Apr 26 07:16 rytec.channels.xml-sat.gz -rwxr--r-- 1 1000 1002 2232 Apr 26 07:16 rytec.channels-greece-en.xml.gz drwxr-xr-x 2 1000 1004 4096 Apr 26 07:12 . -rw-r--r-- 1 1000 1004 137025 Apr 17 00:20
-rwxr--r-- 1 1000 1002 8933 Apr 26 07:16 rytec.channels-norway.xml.gz -rw-r--r-- 1 1000 1002 9400 Apr 26 07:16 rytec.channels-sweden.xml.gz -rwxr-xr-x 1 0 0 41 Apr 10 12:07 index.php -rw-r--r-- 1 1000 1004 107324 Apr 26 07:16 rytec.channels.xml.gz -rwxr-xr-x 1 1000 1004 63858 Apr 26 07:16 rytec.channels-sat.xml.gz drwxr-xr-x 3 0 0 4096 Apr 10 11:41 .. -rw-r--r-- 1 1000 1004 10172 Apr 26 07:16 rytec.channels-slovenia.xml.gz -rwxr--r-- 1 1000 1002 10863 Apr 26 07:16 rytec.channels-finland.xml.gz -rwxr-xr-x 1 1000 1004 1354 Apr 26 07:16 rytec.channels-caledonia.xml.gz -rwxr--r-- 1 1000 1002 47497 Apr 26 07:16 rytec.channels.xml-cable.gz * end parsing directory listing * added directory '/epg_data' to cache ignoring file '/epg_data/.' ignoring file '/epgdata/..' yafc: 'put -n rytec.channels.gz' rytec.channels.gz: no matches found yafc: 'put --newer rytec.channels.gz' rytec.channels.gz: no matches found yafc: 'cache -l' yafc: 'put --newer rytec.channels..gz' rytec.channels._.gz: no matches found yafc: 'ls' ignoring file '/epg_data/.' ignoring file '/epg_data/..'

[doglover3920]

And if I try just one file:

yafc: 'put -n rytec.channels.xml.gz' rytec.channels.xml.gz: no matches found

Willy

[doglover3920]

If I was not clear before.

On FTP servers are no problems. I have the problems on SFTP servers. !!!!!!!

Willy

[doglover3920]

Difference between a SFTP and a FTP server.

A SFTP server:

yafc: 'open kingserver' Looking up rytecepg.kingserver.eu... yafc: 'put -nv /hdd/epg/rytec.channels.gz' /hdd/epg/rytec.channels.gz: no matches found yafc: 'put -nv /hdd/epg/.gz' ** start parsing directory listing -rw-r--r-- 1 1000 1002 5211 Apr 27 05:08 CsatCarabean.channels.xml.gz -rwxr--r-- 1 1000 1002 1357 Apr 27 05:08 Horizons.channels.xml.gz \ end parsing directory listing ***

and a FTP server:

added directory '/epg_data' to cache marked directory '/epg_data' for flush flushed directory '/epgdata' yafc: 'close' Closing down connection... yafc: 'open WoS' <-- [37.187.113.82] 230-OK. Current directory is / <-- [37.187.113.82] 230 95431 Kbytes used (62%) - authorized: 153600 Kb --> [37.187.113.82] PWD <-- [37.187.113.82] 257 "/" is your current location yafc: 'put -nv /hdd/epg/.gz' Initializing passive connection. --> [37.187.113.82] PASV <-- [37.187.113.82] 227 Entering Passive Mode (37,187,113,82,172,98) --> [37.187.113.82] MLSD <-- [37.187.113.82] 150 Accepted data connection <-- [37.187.113.82] 226-Options: -a -l <-- [37.187.113.82] 226 64 matches total _\ start parsing directory listing of '/' *** type=file;size=5211;modify=20150425051103;UNIX.mode=0644;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea019; CsatCarabean.channels.xml.gz type=file;size=1357;modify=20150425050953;UNIX.mode=0744;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea004; Horizons.channels.xml.gz

The SFTP does not provide a year in the file date.

Willy

[doglover3920]

In order to circumvent the problem of not being able to use the -n switch in sftp (maybe just on the server I am trying to communicate with) I have devised a workaround with a linux script.

Request the dir list of the files on the local machine. Also request a dir of the files on the remote. Compares the file sizes and if they are different copy the file to an empty folder on the local machine. Then upload this folder to the remote machine in one go.

This could be easier if yafc would have switch to compare filesizes. Almost all GUI FTP clients have this option. However in yafc this switch is not provided. (Or I cannot find it)

Question: Could a switch be added to upload (or download) files which have different (or smaller, or larger) filesizes?

Willy

[sebastinas]

put now causes the time information to be recorded in a trace. Could you please create a new trace?

[doglover3920]

sftp server and trace recorded:

yafc 1.3.5 trace file started Sun May 10 09:30:53 2015

yafc: 'open kingserver' Looking up rytecepg.kingserver.eu... yafc: 'lcd /hdd/epg' yafc: 'put -n rytec.' ** start parsing directory listing -rwxr-xr-x 1 1000 1004 26297 Apr 28 17:30 rytec.channels-cable.xml.gz -rw-r--r-- 1 1000 1004 59423 May 10 03:37 rytec.channels.xml-sat.gz -rwxr--r-- 1 1000 1002 2232 Apr 28 12:14 rytec.channels-greece-en.xml.gz -rwxr--r-- 1 1000 1002 11866 May 10 03:35 rytec.channels-denmark.xml.gz -rwxr--r-- 1 1000 1002 10919 May 8 15:10 rytec.channels-norway.xml.gz -rw-r--r-- 1 1000 1002 12264 May 8 15:10 rytec.channels-sweden.xml.gz -rwxr--r-- 1 1000 1002 1324 Apr 28 12:14 greek.channels.en.xml.gz -rwxr-xr-x 1 0 0 41 Apr 10 12:07 index.php -rw-r--r-- 1 1000 1004 103619 May 9 09:05 rytec.channels.xml.gz -rwxr-xr-x 1 1000 1004 63858 Apr 28 17:30 rytec.channels-sat.xml.gz -rwxr--r-- 1 1000 1002 1759 Apr 28 12:14 NewCaledonia.channels.xml.gz -rw-r--r-- 1 1000 1004 10216 Apr 28 17:30 rytec.channels-slovenia.xml.gz -rwxr--r-- 1 1000 1002 12670 May 8 15:10 rytec.channels-finland.xml.gz -rwxr-xr-x 1 1000 1004 1354 Apr 28 12:27 rytec.channels-caledonia.xml.gz -rwxr--r-- 1 1000 1002 44239 May 5 03:16 rytec.channels.xml-cable.gz \ end parsing directory listing *** added directory '/epg_data' to cache put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Sun Apr 12 14:04:52 2015

marked directory '/epg_data' for flush put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Sat May 9 19:26:25 2015

put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Sat May 9 10:00:59 2015

put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Sun Apr 12 14:04:52 2015

put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Mon Apr 27 17:29:24 2015

put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Sun May 10 04:05:00 2015

put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Fri May 8 11:59:32 2015

put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Sun May 10 04:05:00 2015

put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Sun Apr 12 14:04:52 2015

put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Fri May 8 11:59:32 2015

put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Sun Apr 12 14:04:52 2015

put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Fri May 8 11:59:32 2015

flushed directory '/epg_data'

Hereafter yafc crashed with a segmentation fault.

Willy

[doglover3920]

And a trace on a ftp server:

xx.xxx.xxx.xx 1.3.5 trace file started Sun May 10 09:37:54 2015

xx.xxx.xxx.xx: 'open WoS' Looking up 37.187.113.82... Connecting to xx.xxx.xxx.xx at port 21... Connected to xx.xxx.xxx.xx ([xx.xxx.xxx.xx]:21). <-- [xx.xxx.xxx.xx] 220---------- Welcome to Pure-FTPd [privsep] [TLS] ---------- <-- [xx.xxx.xxx.xx] 220-You are user number 1 of 5000 allowed. <-- [xx.xxx.xxx.xx] 220-Local time is now 09:42. Server port: 21. <-- [xx.xxx.xxx.xx] 220-This is a private system - No anonymous login <-- [xx.xxx.xxx.xx] 220-IPv6 connections are also welcome on this server. <-- [xx.xxx.xxx.xx] 220 You will be disconnected after 15 minutes of inactivity. remote address: xx.xxx.xxx.xx local address: 192.168.1.19 --> [xx.xxx.xxx.xx] USER ============ <-- [xx.xxx.xxx.xx] 331 User ======= OK. Password required --> [xx.xxx.xxx.xx] PASS **** <-- [xx.xxx.xxx.xx] 230-OK. Current directory is / <-- [xx.xxx.xxx.xx] 230 103175 Kbytes used (67%) - authorized: 153600 Kb --> [xx.xxx.xxx.xx] PWD <-- [xx.xxx.xxx.xx] 257 "/" is your current location xx.xxx.xxx.xx: 'lcd /hdd/epg' xx.xxx.xxx.xx: 'put -n rytec.' Initializing passive connection. --> [xx.xxx.xxx.xx] PASV <-- [xx.xxx.xxx.xx] 227 Entering Passive Mode (37,187,113,82,180,184) --> [xx.xxx.xxx.xx] MLSD <-- [xx.xxx.xxx.xx] 150 Accepted data connection <-- [xx.xxx.xxx.xx] 226-Options: -a -l <-- [xx.xxx.xxx.xx] 226 64 matches total ** start parsing directory listing of '/' * (listing edited) type=cdir;sizd=4096;modify=20150430063824;UNIX.mode=0755;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea026; . type=pdir;sizd=4096;modify=20150430063824;UNIX.mode=0755;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea026; .. type=file;size=14;modify=20150510051002;UNIX.mode=0600;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea027; .ftpquota type=file;size=89;modify=20150408120012;UNIX.mode=0644;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea014; index.htm type=file;size=26297;modify=20150425050937;UNIX.mode=0755;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea002; rytec.channels-cable.xml.gz type=file;size=1354;modify=20150425051144;UNIX.mode=0755;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea02e; rytec.channels-caledonia.xml.gz type=file;size=11866;modify=20150510031947;UNIX.mode=0644;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea005; rytec.channels-denmark.xml.gz type=file;size=12670;modify=20150508151151;UNIX.mode=0644;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea029; rytec.channels-finland.xml.gz type=file;size=2232;modify=20150425051159;UNIX.mode=0744;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea032; rytec.channels-greece-en.xml.gz type=file;size=10919;modify=20150508151151;UNIX.mode=0644;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea030; rytec.channels-norway.xml.gz type=file;size=63858;modify=20150425050959;UNIX.mode=0755;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea009; rytec.channels-sat.xml.gz type=file;size=10216;modify=20150427153722;UNIX.mode=0644;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea00c; rytec.channels-slovenia.xml.gz type=file;size=12264;modify=20150508151206;UNIX.mode=0644;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea035; rytec.channels-sweden.xml.gz type=file;size=44239;modify=20150510032116;UNIX.mode=0644;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea01f; rytec.channels.xml-cable.gz type=file;size=59423;modify=20150510032122;UNIX.mode=0644;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea02b; rytec.channels.xml-sat.gz type=file;size=103619;modify=20150509090951;UNIX.mode=0644;UNIX.uid=1006;UNIX.gid=1006;unique=803g13ea007; rytec.channels.xml.gz * end parsing directory listing *** added directory '/' to cache put -n: remote file: Sat Apr 25 07:09:37 2015 put -n: local file: Sun Apr 12 14:04:52 2015

put -n: remote file: Sun May 10 05:19:47 2015 put -n: local file: Sat May 9 19:26:25 2015

Initializing passive connection. --> [xx.xxx.xxx.xx] PASV <-- [xx.xxx.xxx.xx] 227 Entering Passive Mode (37,187,113,82,144,247) --> [xx.xxx.xxx.xx] TYPE I <-- [xx.xxx.xxx.xx] 200 TYPE is now 8-bit binary --> [xx.xxx.xxx.xx] STOR rytec.channels.xml <-- [xx.xxx.xxx.xx] 150 Accepted data connection marked directory '/' for flush <-- [xx.xxx.xxx.xx] 226-103933 Kbytes used (67%) - authorized: 153600 Kb <-- [xx.xxx.xxx.xx] 226-File successfully transferred <-- [xx.xxx.xxx.xx] 226 2.304 seconds (measured here), 328.95 Kbytes per second put -n: remote file: Sat May 9 11:09:51 2015 put -n: local file: Sat May 9 10:00:59 2015

put -n: remote file: Sat Apr 25 07:09:59 2015 put -n: local file: Sun Apr 12 14:04:52 2015

put -n: remote file: Mon Apr 27 17:37:22 2015 put -n: local file: Mon Apr 27 17:29:24 2015

Initializing passive connection. --> [xx.xxx.xxx.xx] PASV <-- [xx.xxx.xxx.xx] 227 Entering Passive Mode (37,187,113,82,118,253) --> [xx.xxx.xxx.xx] STOR rytec.channels-slovenia.xml <-- [xx.xxx.xxx.xx] 150 Accepted data connection <-- [xx.xxx.xxx.xx] 226-103999 Kbytes used (67%) - authorized: 153600 Kb <-- [xx.xxx.xxx.xx] 226-File successfully transferred <-- [xx.xxx.xxx.xx] 226 0.245 seconds (measured here), 269.52 Kbytes per second put -n: remote file: Sun May 10 05:21:16 2015 put -n: local file: Sun May 10 04:05:00 2015

Initializing passive connection. --> [xx.xxx.xxx.xx] PASV <-- [xx.xxx.xxx.xx] 227 Entering Passive Mode (37,187,113,82,162,194) --> [xx.xxx.xxx.xx] STOR rytec.channels-denmark.xml <-- [xx.xxx.xxx.xx] 150 Accepted data connection <-- [xx.xxx.xxx.xx] 226-104121 Kbytes used (67%) - authorized: 153600 Kb <-- [xx.xxx.xxx.xx] 226-File successfully transferred <-- [xx.xxx.xxx.xx] 226 0.409 seconds (measured here), 298.05 Kbytes per second put -n: remote file: Fri May 8 17:11:51 2015 put -n: local file: Fri May 8 11:59:32 2015

put -n: remote file: Sun May 10 05:21:22 2015 put -n: local file: Sun May 10 04:05:00 2015

put -n: remote file: Sat Apr 25 07:11:44 2015 put -n: local file: Sun Apr 12 14:04:52 2015

put -n: remote file: Fri May 8 17:11:51 2015 put -n: local file: Fri May 8 11:59:32 2015

put -n: remote file: Sat Apr 25 07:11:59 2015 put -n: local file: Sun Apr 12 14:04:52 2015

Initializing passive connection. --> [xx.xxx.xxx.xx] PASV <-- [xx.xxx.xxx.xx] 227 Entering Passive Mode (37,187,113,82,140,23) --> [xx.xxx.xxx.xx] STOR rytec.sources.xml <-- [xx.xxx.xxx.xx] 150 Accepted data connection <-- [xx.xxx.xxx.xx] 226-104140 Kbytes used (67%) - authorized: 153600 Kb <-- [xx.xxx.xxx.xx] 226-File successfully transferred <-- [xx.xxx.xxx.xx] 226 0.104 seconds (measured here), 178.08 Kbytes per second put -n: remote file: Fri May 8 17:12:06 2015 put -n: local file: Fri May 8 11:59:32 2015

flushed directory '/'

• also yafc crashed with a segmentation fault.

[sebastinas]

The crash is now fixed. It seems like your sftp server does not transmit the correct timestamps. I'm not sure we can do anything about that. Current master also traces failures of the sftp_stat call. It'd be interesting to see if there is a useful error message or not.

[doglover3920]

The trace with the latest version:

yafc 1.3.5 trace file started Mon May 11 08:32:35 2015

yafc: 'open kingserver' Looking up rytecepg.kingserver.eu... yafc: 'lcd /hdd/epg' yafc: 'put -n rytec..gz' ** start parsing directory listing -rwxr--r-- 1 1000 1004 124804 May 10 07:32 rytec.channels-denmark.xml -rwxr-xr-x 1 1000 1004 26297 May 10 07:31 rytec.channels-cable.xml.gz -rw-r--r-- 1 1000 1004 59427 May 11 03:24 rytec.channels.xml-sat.gz -rwxr--r-- 1 1000 1002 2232 May 10 07:32 rytec.channels-greece-en.xml.gz drwxr-xr-x 3 1000 1004 4096 May 10 07:32 . -rw-r--r-- 1 1000 1004 137025 Apr 17 00:20
-rwxr--r-- 1 1000 1002 11866 May 10 07:31 rytec.channels-denmark.xml.gz -rwxr--r-- 1 1000 1002 10919 May 10 07:32 rytec.channels-norway.xml.gz -rw-r--r-- 1 1000 1002 12264 May 10 07:32 rytec.channels-sweden.xml.gz -rwxr-xr-x 1 0 0 41 Apr 10 12:07 index.php -rw-r--r-- 1 1000 1004 103631 May 11 03:23 rytec.channels.xml.gz -rwxr-xr-x 1 1000 1004 63858 May 10 07:32 rytec.channels-sat.xml.gz drwxr-xr-x 2 1000 1004 4096 Apr 26 07:47 test -rwxr--r-- 1 1000 1004 776141 May 10 07:46 rytec.channels.xml drwxr-xr-x 3 0 0 4096 Apr 10 11:41 .. -rw-r--r-- 1 1000 1004 10216 May 10 07:32 rytec.channels-slovenia.xml.gz -rwxr--r-- 1 1000 1004 67683 May 10 07:32 rytec.channels-slovenia.xml -rwxr--r-- 1 1000 1002 12670 May 10 07:32 rytec.channels-finland.xml.gz -rwxr-xr-x 1 1000 1004 1354 May 10 07:32 rytec.channels-caledonia.xml.gz -rwxr--r-- 1 1000 1002 44246 May 11 03:24 rytec.channels.xml-cable.gz \ end parsing directory listing *** added directory '/epg_data' to cache Couldn't stat file 'rytec.channels-cable.xml.gz': SFTP server: No such file put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Sun Apr 12 14:04:52 2015

marked directory '/epg_data' for flush Couldn't stat file 'rytec.channels-denmark.xml.gz': SFTP server: No such file put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Sat May 9 19:26:25 2015

Couldn't stat file 'rytec.channels.xml.gz': SFTP server: No such file put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Sun May 10 20:00:38 2015

Couldn't stat file 'rytec.channels-sat.xml.gz': SFTP server: No such file put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Sun Apr 12 14:04:52 2015

Couldn't stat file 'rytec.channels-slovenia.xml.gz': SFTP server: No such file put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Mon Apr 27 17:29:24 2015

Couldn't stat file 'rytec.channels.xml-cable.gz': SFTP server: No such file put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Mon May 11 04:05:00 2015

Couldn't stat file 'rytec.channels-finland.xml.gz': SFTP server: No such file put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Fri May 8 11:59:32 2015

Couldn't stat file 'rytec.channels.xml-sat.gz': SFTP server: No such file put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Mon May 11 04:05:00 2015

Couldn't stat file 'rytec.channels-caledonia.xml.gz': SFTP server: No such file put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Sun Apr 12 14:04:52 2015

Couldn't stat file 'rytec.channels-norway.xml.gz': SFTP server: No such file put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Fri May 8 11:59:32 2015

Couldn't stat file 'rytec.channels-greece-en.xml.gz': SFTP server: No such file put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Sun Apr 12 14:04:52 2015

Couldn't stat file 'rytec.channels-sweden.xml.gz': SFTP server: No such file put -n: remote file: Thu Jan 1 00:59:59 1970 put -n: local file: Fri May 8 11:59:32 2015

flushed directory '/epg_data' yafc: 'close' Closing down connection... yafc: 'exit' Closing down connection...

This bothers me: The file is definitely there!!! Couldn't stat file 'rytec.channels-sweden.xml.gz': SFTP server: No such file

Willy

PS: did you give any thought on compare filesizes?

[sebastinas]

Okay, I'll look into it.

Comparing filesizes does not work if stat does not work. Both filesize and modification time are retrieved with the same function (sftp_stat).

[doglover3920]

I compare filesizes in sh script, after retrieving the dirlist from the local and remote

LOCALFILE=/tmp/local.dir REMOTE=/tmp/remote.dir while read line;do LengteLocal=${line:32:11} FileName=${line:69:40} RemoteLine=$(grep -e "$FileName" $REMOTE) LengteRemote=${RemoteLine:31:11} if [ $LengteLocal -ne $LengteRemote ] ; then cp -f /hdd/epg/$FileName /hdd/epg/tobeuploaded/$FileName fi done < $LOCALFILE

The I just copy the file to an empty dir and upload that dir. The same works with the filedate but it is a bit more complicated.

Willy

[sebastinas]

Well, I mean the code in yafc. Retrieving the file size there suffers from the issue as retrieving the modifiaction time.

[doglover3920]

Yes, I realize that. I just let you know, I am using an alternative way of determining the size and the modification date/time.

About the problem: is this in general a problem with sftp or just this server ? Would it help if I gave you the login parameters? You understand that I cannot publish these here on the open forum.

Willy

[doglover3920]

In filezilla, I'll get the normal view. The information is there.

Willy

[sebastinas]

I hope this is better with f9a33a09b5b79dfe8b374945472bb337a113bfbd.

[doglover3920]

Nope:

yafc 1.3.5 trace file started Wed May 13 16:14:06 2015

yafc: 'lcd /hdd/epg' yafc: 'put -nv rytec..gz' yafc: 'open kingserver' Looking up rytecepg.kingserver.eu... yafc: 'put -nv rytec..gz' * start parsing directory listing * -rwxr-xr-x 1 1000 1004 26297 May 13 14:13 rytec.channels-cable.xml.gz -rw-r--r-- 1 1000 1004 5013022 May 13 03:25 rytecxmltvnordic.gz -rw-r--r-- 1 1000 1004 1579340 May 13 03:23 rytecxmltvosn.gz -rw-r--r-- 1 1000 1004 59474 May 13 14:13 rytec.channels.xml-sat.gz -rwxr--r-- 1 1000 1002 2232 May 13 14:13 rytec.channels-greece-en.xml.gz -rwxr--r-- 1 1000 1002 11987 May 13 14:13 rytec.channels-denmark.xml.gz -rwxr--r-- 1 1000 1002 12025 May 13 14:13 rytec.channels-norway.xml.gz -rw-r--r-- 1 1000 1002 13784 May 13 14:13 rytec.channels-sweden.xml.gz -rwxr--r-- 1 1000 1002 1324 Apr 28 12:14 greek.channels.en.xml.gz -rw-r--r-- 1 1000 1004 103677 May 13 14:13 rytec.channels.xml.gz -rwxr-xr-x 1 1000 1004 63858 May 13 14:13 rytec.channels-sat.xml.gz -rw-r--r-- 1 1000 1004 10216 May 13 14:13 rytec.channels-slovenia.xml.gz -rwxr--r-- 1 1000 1004 67683 May 10 07:32 rytec.channels-slovenia.xml -rwxr--r-- 1 1000 1002 12567 May 13 14:13 rytec.channels-finland.xml.gz -rwxr-xr-x 1 1000 1004 1354 May 13 14:13 rytec.channels-caledonia.xml.gz -rwxr--r-- 1 1000 1002 44246 May 13 14:13 rytec.channels.xml-cable.gz * end parsing directory listing * added directory '/epg_data' to cache marked directory '/epg_data' for flush flushed directory '/epg_data'

And it uploads every file.

Willy

[doglover3920]

Sorry did not compile your latest version: After using this latest, I got this:

yafc> open kingserver
yafc rytecepg.kingserver.eu:/epg_data> lcd /hdd/epg
local working directory is '/media/hdd/epg'
yafc rytecepg.kingserver.eu:/epg_data> put -n rytec.*.gz
Remote file 'rytec.channels-cable.xml.gz' is newer than local, skipping...
Remote file 'rytec.channels-denmark.xml.gz' is newer than local, skipping...
Remote file 'rytec.channels.xml.gz' is newer than local, skipping...
Remote file 'rytec.channels-sat.xml.gz' is newer than local, skipping...
Remote file 'rytec.channels-slovenia.xml.gz' is newer than local, skipping... Remote file 'rytec.channels.xml-cable.gz' is newer than local, skipping...
Remote file 'rytec.channels-finland.xml.gz' is newer than local, skipping...
Remote file 'rytec.channels.xml-sat.gz' is newer than local, skipping...
Remote file 'rytec.channels-caledonia.xml.gz' is newer than local, skipping... Remote file 'rytec.channels-norway.xml.gz' is newer than local, skipping...
Remote file 'rytec.channels-greece-en.xml.gz' is newer than local, skipping... Remote file 'rytec.channels-sweden.xml.gz' is newer than local, skipping...
yafc rytecepg.kingserver.eu:/epg_data>

And from the trace:

put -n: remote file: Wed May 13 16:14:54 2015 put -n: local file: Sun Apr 12 14:04:52 2015

put -n: remote file: Wed May 13 16:14:54 2015 put -n: local file: Tue May 12 08:39:26 2015

put -n: remote file: Wed May 13 16:14:55 2015 put -n: local file: Tue May 12 09:33:23 2015

put -n: remote file: Wed May 13 16:14:55 2015 put -n: local file: Sun Apr 12 14:04:52 2015

It looks like you got it now.

Willy

[sebastinas]

Great!