Destroy s3 feed issue with feedSubPath

dplatonovTM commented 10 months ago

Problem: When using a feedSubPath in Sleet to create a feed in an empty bucket, problems arise when executing the destroy command, only the index.json file is deleted in the feedSubPath

Steps to reproduce:

I create a feed in an empty bucket without using a feedSubPath.
The feed and files are created correctly.
When you execute the destroy command, the bucket is completely cleared (even files and directories not created by the feed are deleted).
However, when I start using feedSubPath, Destroy stops deleting files in the feedSubPath directory, except for index.json.

Expected Behavior: I expect that when using feedSubPath, Destroy will also delete all files in the feedSubPath directory, and not just index.json.

Additional Information: Sleet lists the files to delete in the console output, but only deletes index.json.

emgarten commented 10 months ago

Are you able to put all sub feeds at the same directory level?

Having nested feeds in this way is a known issue and destroy is not supported in this scenario.

Potentially this could be solved by keeping an index file at the root of the bucket so that all feeds can discover that there are multiple feeds present, or a nesting. With the current design each feed only knows about its own files.

dplatonovTM commented 10 months ago

The thing is that I only have one feed. If I place the index file and all other files in the root directory, everything works correctly. If I try to put the index file and the rest of the feed files into a directory, the destroy stops working for everything except the index file itself. Looks like feed using subPath cannot even delete its own files.

emgarten commented 10 months ago

Looks like feed using subPath cannot even delete its own files.

That does sound like a bug then. A sub feed should be able to delete itself.

dplatonovTM commented 10 months ago

Hello. Can I somehow help fix this? Maybe I should demonstrate how I reproduce this?

emgarten commented 10 months ago

Are you able to run sleet in the debugger while you destroy a feed with a sub path?

I would set a breakpoint here: https://github.com/emgarten/Sleet/blob/704fe023e597504764df84330b689ba239bc805d/src/SleetLib/FileSystem/AmazonS3FileSystem.cs#L81

Destroy lists all files in the feed, then deletes them. My guess is that it isn't finding the other files. Looking at the code it should work, but there must be an issue with it.

dplatonovTM commented 10 months ago

Unfortunately no, I am not familiar with debugging tools for Dotnet applications. I tried but couldn't get anything useful out of it. But running the destroy command in verbose mode showed that files that are not deleted are skipped.

emgarten commented 10 months ago

Does the config have a bucket of sleet and also a feedsubpath of sleet?

Does https://bucket.s3.eu-central-1.amazonaws.com/sleet/sleet/sleet.packageindex.json get deleted?

dplatonovTM commented 10 months ago

There is my config (I usually use EC2 instance profile instead of keys): {

  "sources": [
    {
      "name": "cachedS3Feed",
      "type": "s3",
      "path": "https://bucket.s3.eu-central-1.amazonaws.com/sleet",
      "feedSubPath": "/sleet/",
      "bucketName": "bucket",
      "region": "eu-central-1",
      "accessKeyId": "123",
      "secretAccessKey": "456"
    }
  ]
}

Nope. https://bucket.s3.eu-central-1.amazonaws.com/sleet/sleet/sleet.packageindex.json was skipped and not deleted.

emgarten commented 10 months ago

Does it work if you change /sleet/ to sleet?

dplatonovTM commented 10 months ago

Same. I tried "sleet", "/sleet", "/sleet/", "sleet/". Index.json was deleted only.

emgarten commented 10 months ago

@dplatonovTM this has been fixed in https://www.nuget.org/packages/Sleet/5.2.11

Let me know if you hit any other issues with it.

I've updated the functional test to verify all files get removed as part of a sub feed on AWS S3. This issue was specific to S3.

emgarten / Sleet

Destroy s3 feed issue with feedSubPath #185