SFTP uploading taking lot of time

[DorvakOff]

Hello, the uploading of files is taking too much time (4min39s) for only 45mb

  deploy:
    name: Deploying to the server
    runs-on: ubuntu-latest
    needs: [ compile-jar, update-version ]
    steps:
      - name: Restore the stored App from artifacts
        uses: actions/download-artifact@v2
        with:
          path: build
      - name: Moving files to root
        run: mv build/App/* ./
      - name: Clearing extras
        run: rm -r build
      - name: Listing files
        run: tree
      - name: SFTP Deploy
        uses: wangyucode/sftp-upload-action@v1.4.5
        with:
          host: ${{ secrets.SFTP_ADDRESS }}
          username: ${{ secrets.SFTP_USER }}
          password: ${{ secrets.SFTP_PWD }}
          port: 2022
          localDir: './'
          remoteDir: './'
          dryRun: false

[Elikill58]

I've same problem. With multiple files it's long.

Maybe zip files, then upload, then unzip can help ?

Took more 3m for me:

[wangyucode]

TODO

• [ ] Add an optional parameter zipBeforeUpload, to speed up the upload of many small files.

[DorvakOff]

Actually it's only a "small" file that is taking a lot of time 😕

[ncgautomate]

my deployment taking like 30 minutes. Every time i push the code, it is trying to upload the same files. I have thousands of files. and it taking more than 30 minutes. I have used the parameter forceUpload: false

I tried to add the optional parameter, but got warning https://prnt.sc/5jqqSuD2OJkd can you add this parameter in the main branch?

any suggestions?

[ncgautomate]

each time I do PR, when the action runs, it keep pushing / uploading the existing files to SFTP server, even those files already exist. the app thinking each time i modify a single file and push, it thinks all of files in the app directory are new and trying to deploy them again. it's takes about 40 minutes to upload all those small files.

https://prnt.sc/Ty6eBNCtPjjS

how do i say only upload files are modified or newer?

[Amerlander]

I have also set forceUpload: false but it uploads all files (the log says "uploading newer file:"…)

To minimize downtime, I thought about also uploading all files to a tmp folder, then move all existing files (except the ones in exclude) to another tmp and then move the new files in place and then delete the old tmp files. I think moving is often faster than deleting, and that could lead to a very short downtime.

Edit: my file sync is currently 11m for ~80mb and ~1000 files in ~100 folders.

[wangyucode]

Regarding performance improvement, although uploading files is currently asynchronous, comparison files are synchronous. TODO:

• [ ] it is possible to read all remote and local files asynchronously to speed up the comparison process.

[wangyucode]

Regarding performance improvement, although uploading files is currently asynchronous, comparison files are synchronous. TODO:

• [x] it is possible to read all remote and local files asynchronously to speed up the comparison process.

Refactored in v2.0.0

[ApplecakeYT]

Can you make it so it stores like a sync-state file on the remote server so that the action already knows which files it has read remotely and doesn't have to reread files. This could save lots of time, especially on larger builds..

Something similar is done for FTP Deploy on https://github.com/SamKirkland/FTP-Deploy-Action for example. It stores a list in the remote server and only updates files that were changed.

[SedlarDavid]

Can you make it so it stores like a sync-state file on the remote server so that the action already knows which files it has read remotely and doesn't have to reread files. This could save lots of time, especially on larger builds..

Something similar is done for FTP Deploy on https://github.com/SamKirkland/FTP-Deploy-Action for example. It stores a list in the remote server and only updates files that were changed.

Feature like this will be much helpful since upload to limited server, without ssh access, when there is need for uploading around 10k project and dependencies files takes around hour.

[wangyucode]

If you have many files to upload, packaging them into tar.gz and notifying the server to download and decompress might be a better choice. Please refer to my new project, which is written in Rust, very lightweight and fast, and doesn't require a server key/password. https://github.com/wangyucode/crane