FTP transport moves files insted of copy them

[sahami]

I set up wimleers-fileconveyor-8ab0943 on my server. Here is my cnfiguration file

config>    sources>     source name="Drupal_Files" scanPath="/test" documentRoot="/test" />   /sources>   servers>         server name="LB1" transporter="ftp" maxConnections="5">                 host>10.10.10.2 /host>                 username>user /username>                 password>password /password>                 url>http://localhost/daemontest/ /url>         /server>   /servers>          rules>                 rule for="Drupal_Files" label="Copy of website files on Web1">                         destinations>                                 destination server="LB1" path="test" />                         /destinations>                 /rule>       /rules> /config>

It copies new and modified files but "automatically deletes them" after sync as you see below.

2012-01-27 23:22:12,318 - Arbitrator.FSMonitor - INFO - Detected 'CREATED' event for '/test/t15.x' through inotify (for monitored path '/test'). FSMONITOR CALLBACK FIRED: input_file='/test/t15.x' event=1" discovered_through=inotify 2012-01-27 23:22:12,318 - Arbitrator.FSMonitor - INFO - Detected 'MODIFIED' event for '/test/t15.x' through inotify (for monitored path '/test'). FSMONITOR CALLBACK FIRED: input_file='/test/t15.x' event=2" discovered_through=inotify 2012-01-27 23:22:12,496 - Arbitrator - INFO - Discover queue -> pipeline queue: '/test/t15.x'. 2012-01-27 23:22:12,584 - Arbitrator - INFO - Pipeline queue: merged events for '/test/t15.x': CREATED + MODIFIED = CREATED. 2012-01-27 23:22:12,585 - Arbitrator - INFO - Discover queue -> pipeline queue: '/test/t15.x'. 2012-01-27 23:22:12,816 - Arbitrator - INFO - Pipeline queue -> filter queue: '/test/t15.x'. 2012-01-27 23:22:12,816 - Arbitrator - INFO - Filtering: '/test/t15.x' matches the 'Copy of website files on Web1' rule for the 'Drupal_Files' source! 2012-01-27 23:22:12,816 - Arbitrator - INFO - Filter queue -> transport queue: '/test/t15.x' (rule: 'Copy of website files on Web1'). 2012-01-27 23:22:12,817 - Arbitrator - INFO - Transport queue: '/test/t15.x' to transfer to server 'LB1' with transporter #1 (of 1), place 1 in the queue. TRANSPORTER CALLBACK FIRED: (curried): input_file='/test/t15.x' (curried): event=1 (curried): rule='Copy of website files on Web1' (curried): processed_for_server='None' output_file='/test/t15.x' transported_file='test/t15.x' url='http://localhost/daemontest/test/t15.x' server='LB1' 2012-01-27 23:22:13,007 - Arbitrator - INFO - Transport queue -> DB queue: '/test/t15.x' (server: 'LB1'). 2012-01-27 23:22:13,124 - Arbitrator - WARNING - Deleted '/test/t15.x' as per the 'Copy of website files on Web1' rule. 2012-01-27 23:22:13,124 - Arbitrator.FSMonitor - INFO - Detected 'DELETED' event for '/test/t15.x' through inotify (for monitored path '/test'). FSMONITOR CALLBACK FIRED: input_file='/test/t15.x' event=4" discovered_through=inotify 2012-01-27 23:22:13,201 - Arbitrator - WARNING - Synced: '/test/t15.x' (CREATED). 2012-01-27 23:22:13,520 - Arbitrator - INFO - Discover queue -> pipeline queue: '/test/t15.x'. 2012-01-27 23:22:13,762 - Arbitrator - INFO - Pipeline queue -> filter queue: '/test/t15.x'. 2012-01-27 23:22:13,762 - Arbitrator - INFO - Filtering: '/test/t15.x' matches the 'Copy of website files on Web1' rule for the 'Drupal_Files' source! 2012-01-27 23:22:13,763 - Arbitrator - INFO - Filtering: queued transporter to server 'LB1' for file '/test/t15.x' to delete it ('Copy of website files on Web1' rule). 2012-01-27 23:22:13,763 - Arbitrator - INFO - Transport queue: '/test/t15.x' to transfer to server 'LB1' with transporter #1 (of 1), place 1 in the queue. TRANSPORTER CALLBACK FIRED: (curried): input_file='/test/t15.x' (curried): event=4 (curried): rule='Copy of website files on Web1' (curried): processed_for_server='None' output_file='/test/t15.x' transported_file='test/t15.x' url='None' server='LB1' 2012-01-27 23:22:14,010 - Arbitrator - INFO - Transport queue -> DB queue: '/test/t15.x' (server: 'LB1'). 2012-01-27 23:22:14,400 - Arbitrator - WARNING - Synced: '/test/t15.x' (DELETED).

As i understand fileconveyor does not move files from source and copies them. But I downt know why it moves them at least in ftp transporter.

Thanks M. Sahami

[stongo]

The FTP transporter removes all original files on the server for me, as well. I ran file conveyer on my dev server to sync files to my CDN thankfully, otherwise this would have toasted my production site. This is a critical bug.

[kneedrag]

Same problem here using Mosso and symlink_or_copy transporters. FileConveyor deletes all the source files. A workaround is to set the fileDeletionDelayAfterSync to an extremely high number that hopefully would never be hit. Would be better to find out why this is occurring and fix it.

This workaround seems to be working for me: rule for="items" label="cdn" fileDeletionDelayAfterSync="60000000000000000"

[satheshf12000]

@sahami @stongo @kneedrag Any updates guys? My files are deleted as well.. Lucky that I did on my Test server as well..

[stongo]

Sorry no progress from me. I gave up on file conveyor and was able to switch my cdn account to origin pull.

[kneedrag]

add to your rule:

fileDeletionDelayAfterSync="60000000000000000"

Then periodically use sqlite3 to open persistent_data_db and:

delete from files_to_delete_list where id > 0 ;

Good luck.

[kennywyland]

Wow, I'm glad I tar.gz'd my files before starting my "copy"... I shut it down as soon as I saw it was deleting files. I'm using the mosso transporter for Rackspace CloudFiles. Not only was it deleting my files on my source host, but those files do not appear to be ANYWHERE on my CloudFiles container. Good times. So if I hadn't tar.gz'd them, they'd be gone forever.

[wimleers]

Fixed over at #108.

Sorry for introducing such a terrible bug :(