Can't find my S3 assets?

[DerHerzog7]

The module works fine for files in my user data, but can't seem to find the files in my S3 set up. Other modules don't have this issue... Is there a bug?

[Aedif]

Have you added the path to the s3 folder in the 'Variant art search paths' setting?

The format should be as so: s3:{bucket-name}:{path}

e.g. some/other/path/ another/user/data/path/ s3:ArtBucket:token/art/path/

[DerHerzog7]

Yep! I've even tried different variations of it, just in case.

[DerHerzog7]

any idea what's going on here?

[jbowensii]

The S3 bucket is hard to setup, through trial and error this is the correct format (the tip on the setup screen did not work for me) "s3:name_of_my_bucket:root_directory/Portraits/"

The name of the S3 bucket does NOT contain its location or any other data from say a URL, then a colon (required) followed by a directory path (any depth and name should work), MOST IMPORTANT end with a "/" at least on LINUX.

[Envaris-Avalon]

I found that when I had not many assets in my S3 bucket everything was fine. However, once I added all my assets to S3 (7519 files) + the Caeora assets (both patreon support and free). That's why it never finishes caching and it just doesn't work.

Is there limit to the number of files? Or depth to the folder structure?

My tokens are from different sources each with their on folder structures that came with them.

[Aedif]

I do not use S3 storage myself and had only set it up for testing purposes when I originally implemented support for it. So unfortunately at the moment issues with S3 are not something I can test.

If anyone's comfortable with running stuff in the console this is how the module queries for S3 resources:

e.g.

await FilePicker.browse('s3', "TopDownTokens\", {
        bucket: "tokens",
      })

Which should return something like this:

{
    "target": "TopDownTokens",
    "private": false,
    "gridSize": null,
    "dirs": [
        "TopDownTokens/TP13Pirates",
        "TopDownTokens/TP17HairyOrcsGoblins",
    ],
    "privateDirs": [],
    "files": [
         "dog_hound.png",
         "female_blacksmith_hammer.png"
    ],
    "extensions": []
}

The module stores all the "files" that have been returned, and then performs the query again, this time using the "dirs" as the new paths. So for example the next query would be:

await FilePicker.browse('s3', "TopDownTokens/TP13Pirates/", {
        bucket: "tokens",
      })

The module repeats this until all the files have been retrieved; stopping when no more "dirs" have been found or if "target" is returned as "."

This is the condensed down version of what the modules does. Run this in the console and you should see the list of all the files found in your S3 storage.

async function test_s3(path, bucket) {
  let foundImages = [];

  let files = await FilePicker.browse('s3', path, {
    bucket: bucket,
  });

  for (let image of files.files) {
    foundImages.push(image);
  }

  if (files.target == '.') return [];

  for (let dir of files.dirs) {
    foundImages = foundImages.concat(await test_s3(dir, bucket));
  }
  return foundImages;
}

await test_s3("path/in/the/bucket", "bucket_name")

I'd be curious to know if it takes a long time to run or if it throws up any interesting errors.

[Envaris-Avalon]

No errors were thrown. Interestingly, although I thinking nothing has changed it has managed to crunch through 8400 assets. Rather than giving the entire directory I've started splitting the "big" directory apart and adding each to Variant Token for processing and caching.

[Aedif]

If there are still people having issues with S3 buckets,, the module as of release 2.6.0 now will bring up a FilePicker when selecting the folder icon, allowing you select a folder within an S3 bucket and have search path automatically populate with the correct format.

Should make the setup a bit easier.

[christianhturner]

I attempted to go down to more specific folders i.e. s3:bucket-name:Tokens/Abberations (no more than 100 images in this folder) I'm still receiving a pop up that the directory doesn't exist. Even with the file picker, It'll let me navigate to my S3 bucket and select it, but the module is still throwing an error stating that the directory doesn't exist.

This was actually working for me until I tried this morning and Variant art was no longer working. I'm curious to whether or not another module could be causing issues. I also ran the commands above and it queried instantly and revealed all of my directories as well.

await FilePicker.browse( 's3', "Tokens/", { bucket: "foundry-chimis-vtt-assets" }) Object { target: "Tokens/", private: false, gridSize: null, dirs: (42) […], privateDirs: [], files: (1) […], extensions: [], bucket: "foundry-chimis-vtt-assets" } ​ bucket: "foundry-chimis-vtt-assets" ​ dirs: Array(42) [ "Tokens/1930's Characters", "Tokens/Aberration", "Tokens/Age_of_Ashes_Tokens", … ] ​​ 0: "Tokens/1930's Characters" ​​ 1: "Tokens/Aberration" ​​ 2: "Tokens/Age_of_Ashes_Tokens" ​​ 3: "Tokens/Animals" ​​ 4: "Tokens/Arctic Animals" ​​ 5: "Tokens/Astral" ​​ 6: "Tokens/Boss Monster Token Set 1" ​​ 7: "Tokens/CC1 Aatxe - Cueyatl Hi" ​​ 8: "Tokens/CC1 Aatxe - Cueyatl Lo" ​​ 9: "Tokens/CC2 Hi" ​​ 10: "Tokens/CC2 Lo" ​​ 11: "Tokens/Constructs" ​​ 12: "Tokens/Corpse" ​​ 13: "Tokens/Cthulu" ​​ 14: "Tokens/Cyborgs & Soldiers" ​​ 15: "Tokens/Demons_Devils" ​​ 16: "Tokens/Dessert" ​​ 17: "Tokens/Diety" ​​ 18: "Tokens/Dinosaur" ​​ 19: "Tokens/Dragons" ​​ 20: "Tokens/Elemental" ​​ 21: "Tokens/FA_Tokens_Webp" ​​ 22: "Tokens/Fey" ​​ 23: "Tokens/Fiends" ​​ 24: "Tokens/Heroes" ​​ 25: "Tokens/Humanoid" ​​ 26: "Tokens/Incorporeal" ​​ 27: "Tokens/Magical Beast" ​​ 28: "Tokens/Monstrosity" ​​ 29: "Tokens/Other" ​​ 30: "Tokens/Pathfinder Icons 2" ​​ 31: "Tokens/Pathfinder Icons" ​​ 32: "Tokens/Sci-Fi Token Pack 002" ​​ 33: "Tokens/Sci-Fi Token Pack 003 - Athey's Aliens" ​​ 34: "Tokens/ShadowUmbalCreatures" ​​ 35: "Tokens/Space Opera" ​​ 36: "Tokens/Tome of Beasts Pack 1" ​​ 37: "Tokens/Tome of Beasts Pack 2" ​​ 38: "Tokens/Tome of Beasts Pack 3" ​​ 39: "Tokens/Undead" ​​ 40: "Tokens/Vehicle" ​​ 41: "Tokens/familiars_summons" ​​ length: 42 ​​

: Array [] ​ extensions: Array [] ​ files: Array [ "https://foundry-chimis-vtt-assets.s3.amazonaws.com/Tokens/Mercenary.png" ] ​ gridSize: null ​ private: false ​ privateDirs: Array [] ​ target: "Tokens/" ​ : Object { … }

[christianhturner]

Error I receive after selecting the S3 bucket Uncaught (in promise) The specified bucket does not exist foundry.js:45756:3 browse foundry.js:45756 AsyncFunctionThrow self-hosted:640 (Async: async) render foundry.js:45892 _onBrowseFolder configureSettings.js:301 _onBrowseFolder self-hosted:1124 jQuery 8 dispatch handle (Async: EventListener.handleEvent) add Ee each each Ee on activateListeners configureSettings.js:153 _render foundry.js:3090 AsyncFunctionNext self-hosted:635 (Async: async) _render foundry.js:3764 render foundry.js:3021 onClickSubmenuWrapper colorSetting.js:20 call_wrapper libWrapper-wrapper.js:595 _onClickSubmenu#0 libWrapper-wrapper.js:189 _onClickSubmenu#0 self-hosted:1124 jQuery 2 dispatch handle

[Aedif]

I think I found the issue. I'll push a fix out in the next release.

[Aedif]

Alright the S3 path parsing should be fixed in release 2.10.0. Update when you can and inform me if there are still problems.

[christianhturner]

The update worked beautifully for me. I also had no issues caching the root directory for my tokens. It successfully cached 18,000 images :). That does take a while, but that is not the plugins fault. Speed will likely depend mostly on your hardware and javascript. Should someone run into an issue with large file sets needing to be cached make sure you utilize the feature provided by Aedif under misc. > enable 'Static Cache'.

If you have set your S3 prior to the patch I would delete the path and re add it and manually cache the data under misc. > cache image button. At least for me, it wouldn't trigger on it's own. I'm assuming that is likely due to the fact that the path was pre-existing and I had static cache enabled previously. I was receiving an error that no file could be found (which is to be expected) after the update.

[Aedif]

Issues quotes about the static cache should be resolved as of release 2.13.0

S3 problems as well I believe are solved for now, so I will be closing the issue. If anyone responds thinking they still persist I'll re-open it.