Sync function running infinitely and calling abort does not stop the execution of the sync function

[fullStackDev316]

We had an issue using this library where the sync function never returned even after 2 days of execution. We have a half uploaded file on the S3 bucket but there seems to be no progress in the last day and a half.

This led us look into using abort operation in case the promise does not resolve. But even using the code sample provided, we are not able to kill the sync operation after 10 sec. We know this as we are seeing the size and count metrics begin printed on the console.

Is there something that we are missing? Should the library eventually timeout?

Any help would be appreciated. Please let me know if you have any additional questions. Thanks in advance!

const EventEmitter = require('events');
const { TransferMonitor } = require('s3-sync-client');

const monitor = new TransferMonitor();
monitor.on('progress', (progress) => console.log(progress));
setTimeout(() => monitor.abort(), 10000); // optional abort

await sync('s3://mybucket', '/path/to/local/dir', { monitor });

/* output:
...
{
  size: { current: 11925, total: 35688 },
  count: { current: 3974, total: 10000 }
}
...
and aborts unfinished sync after 10s (promise rejected with an AbortError) 
*/

// to pull status info occasionally only, use monitor.getStatus():
const timeout = setInterval(() => console.log(monitor.getStatus()), 2000);
try {
    await sync('s3://mybucket', '/path/to/local/dir', { monitor });
} finally {
    clearInterval(timeout);
}

[jeanbmar]

Hey there and thank you for opening an issue! I was able to reproduce the problem with a multipart upload, I will fix it this week-end.

[jeanbmar]

Sorry for the extra delay, it's now fixed in https://github.com/jeanbmar/s3-sync-client/pull/35. Corresponding test: https://github.com/jeanbmar/s3-sync-client/blob/cdb8bf7b7c2f3740d3c252f5eaef957f2e2126be/test/s3-sync-client.test.js#L228-L239

[ChristopheBougere]

Hi @jeanbmar I'm seeing a very similar issue. I'm syncing from local (EFS) to S3, and sometimes I have a timeout (in lambda after 15 minutes). I added a monitor to see what's happening, but during all the 15 minutes I'm only seeing this:

It looks like nothing has to be synced, but the sync method doesn't end. Any idea?

[jeanbmar]

Hello @ChristopheBougere,

Does the Lambda always timeout when you use TransferMonitor this way? This 0 everywhere thing is definitely weird and I wonder if this specific timeout can come from setInterval. Timeout without logs being still an issue of course. Are you syncing small files, big files (>5MB) , both? How many files can be synced? What sync options are you using? Could you share some code ideally? EFS has a throughput limitation that can prevent data transfers when reached, can you confirm the limitation is not reached?

[ChristopheBougere]

Thanks for your answer!

Does the Lambda always timeout when you use TransferMonitor this way?

No, only for a few invocations. By the way my sync function is invoked every 5 seconds in a stepfunction (so quite frequently).

I wonder if this specific timeout can come from setInterval. Timeout without logs being still an issue of course.

So the timeouts were already happening before I add the monitor. I added the monitor to try to understand what was happening.

Are you syncing small files, big files (>5MB) , both?

In this case small files (0.5MB)

How many files can be synced?

Only 24 files, which are added over time.

What sync options are you using? Could you share some code ideally?

Sure, here is a snippet:

import { TransferMonitor, S3SyncClient } from 's3-sync-client';

const client = new S3Client({
  region: process.env.AWS_REGION,
  useAccelerateEndpoint: true,
});
const syncClient = new S3SyncClient({ client });

const monitor = new TransferMonitor();
const timeout = setInterval(() => console.log(monitor.getStatus()), 2000);
await syncClient.sync(`/mnt/efs/subfolder/${resourceId}/results`, `s3://${BUCKET_NAME}/subfolder/${resourceId}/results`);
clearInterval(timeout);

EFS has a throughput limitation that can prevent data transfers when reached, can you confirm the limitation is not reached?

That's a good lead, let me come back to you tomorrow with more information about that! It would explain the fact it only occurs sometimes (when the EFS reaches limits in another process).

[ChristopheBougere]

Hi @jeanbmar I can confirm we're regularly reaching throughput limits:

However, I doubt that this would lead to 15 minutes timeouts. I wonder if something isn't well caught in s3-sync-client and leads to the timeout. Meanwhile, I'll implement aborting after a few seconds and retry on my side.

[jeanbmar]

@ChristopheBougere, I agree, I don't think the throughput limitation is the issue here. I'll open a new issue to track this.