Pull assets from AWS S3 bucket to nextjs project

[ihsanashi]

• [x] Create local postgres database to host assets (tables for photos, stories & videos)
• [x] Use AWS CLI to get all objects in the S3 bucket
• [x] Use jq to remove undesired keys from each JSON file returned from running the previous AWS CLI commands (should only be left with Key and LastModified keys)
• [x] Use jq to transform JSON data from the previous step, to create a new JSON file with path and last_modified_at keys. For path, prefix with {AWS_REGION}/{AWS_BUCKET_NAME}. Maybe this part can be done when merging the modified AWS JSON with Instagram metadata JSON files. Transformed the keys first.
• [x] Use jq to combine data from AWS CLI with metadata JSON files from Instagram to the correct format (accounting for alt text, caption and URL paths). The common key/value pair is the path.
• [x] Use jq to insert a new key for AWS & cloudinary URLs in the merged JSON file
• [x] Generate INSERT query for each object to run in postgres Supabase

[ihsanashi]

Command for getting all the filenames from a particular folder in an S3 bucket, and saving them to a local file: aws s3api list-objects --bucket your-bucket-name --prefix "example/sub/directory/" > /Users/example/folder/sub/folder/filenames.json

Reference

Original JSON dump from AWS includes the root object for each folder, eg. /instagram/stories, so those were deleted. I also have uploaded metadata files for each folder (photos, stories & videos) to S3 as a backup, these were also included in the JSON dump, and manually deleted in the local JSON file. Also, I removed the first key whose values is the array of objects in each JSON file, eg. "stories" key in stories.json file, to ensure the file is just an array of objects, with no parent key referring to said array.

Then I use jq to manipulate the JSON files to only have the key value pairs that I need.

[ihsanashi]

Supabase schema