work with s3 buckets

[bbelderbos]

@abaddonpuff can you flesh out the use case here? Are we posting to or pulling from a bucket? And to what end?

Starter code for upload: https://github.com/PyBites-Open-Source/pybites-tools/blob/main/pybites_tools/aws.py

[bbelderbos]

@abaddonpuff we can go input or output, I think input is more interesting for this app.

What do you think about for example:

• create a bucket
• set up a trigger
• if a log file comes in
• an AWS lambda function gets called
• that calls a new API endpoint on your app (when deployed) pushing the file content
• app parses it (your analyze log script) and shows a graph in a new view

Only concern is unencrypted data going over the wire (aws -> your app), but we'd prototype it with fake data anyways.

Something like this or did you have another use case in mind?

[bbelderbos]

@abaddonpuff thoughts? And any other progress?

[abaddonpuff]

Sorry Bob, had to stay extra late yesterday at work because of some investigations and it was super difficult for me to make progress. Will make up for it tonight and tomorrow morning.

I figured an idea that would be fun.

• I could have my app with different tabs of my most common things (League of legends, X, Amazon Shopping, LinkedIn, Spotify) - basically grabing just a couple things I could use an API for (and anyone could create a free API key for) and simulating a multi-task centralized application e.g. doing different stuff in a single web app)
• All of this daily activity on these apps would be the triggers. (New game, new posts, new price on product or discount, new job announced, new song for an artist)
• Standardize a history of activity that happened as a single log structure (e.g. bbelderbos posted just text, bbelderbos posted an image, bbelderbos posted a link, bbelderbos created a space) and send to S3 for extended storage.
• Grab the s3 storage information and use it to create dashboards depending on the type of data e.g. League of Legends could get amount of wins vs Loses. Amazon shopping could track the history of a price for a product, LinkedIn could get the salary ranges for jobs for an interesting org. Spotify could get me my most listened songs.
• Then put everything in a container for portability.

Sorry I think I used most of my night today to figure out all of this (sometimes the ideas I get involves my job, the things I know of and they have so many dependencies to things that I don't have or that cost). So I tried to change my mentality to the things I can control and are free and I like and for the 1st application this would be pretty much the same structure, just different apis, different solutions and I can easily extrapolate the knowledge to the tools I have on my work.

This idea has also a lot more structure to it and it will be fun to learn.

It took me more than expected to figure out this so I will start the coding part tomorrow and make up for the time lost tomorrow morning and over the weekend too.

Pasting here the Design

[bbelderbos]

Nice, this is way more practical and it'll be fun! I agree that work dependencies have slowed us down a bit (although you still have progressed + learned a lot I think). So let's kick this off as a new project, we'll discuss in our meeting later today ...