Limit the projects we store in the DB and drop older entries?

[DavidReider]

Currently, the project makes use of the free tier of Supabase, which is a database that has Up to 500MB of storage.

If we store hourly data of all projects on Magic Eden, this will fill quickly.

Maybe we only take select projects and/or top x% of traded projects so help limit this.

Also, we can potentially drop older data? Maybe we only want to keep a weeks worth of data for a given project?

[WaterlessPiano2]

how about we only fetch data daily for now instead of hourly? I like the idea of only working with some collections for now. I know Solana Monkey Business have a lot of derivatives that end with the word 'Business', so we could use them for now.

I remember you wanted to show the images of the collections, for that we need to save the image location near the collection name. The name and image URL is the biggest strings in the table and they are not going to be changed often, I think we can save them in a separate table called 'collection_metadata', so we don't repeat this information the 'SolanaFloorTracker' table.

we can have an id as primary key in 'collection_metadata' table and same data as foreign key as collection_id in 'SolanaFloorTracker' table. if each collection has a unique ID in the magic eden API. we could also save this in the 'collection_metadata'. So we can use this magic_eden_Id to update the collection name or image URL.

What do you think?

[WaterlessPiano2]

Also by having a unique ID per project, we can have a more robust way to combine data in the front end. We would then rely on the ID's rather than the order of results like we do now.

[DavidReider]

Hi @WaterlessPiano2 !

We can definitely swap it to fetch data daily for now instead of hourly, that would help with the increasing database size (once we get the api.js running consistently).

Additionally, we can also change it to use the 'Business' derivatives, we would just need to adjust the logic for that. Getting the images wasn't super important - it was more of a 'want' than a need, but I do like the idea of a secondary table for that.

I'd have to check the JSON of the magic eden API to see if the projects have a unique ID to them, but I also think the names are unique so it may just use that (I don't think there can be two "Solana Monkey Business" projects).

In the end, we would probably still want to drop old data (or I'd have to purchase a bigger storage plan, but without a source of income on the project I am not sure I would be interested in spending a ton on a better plan currently) and we would also probably still want to only track projects that are gaining traction or are already decently sized. Probably no point in tracking a project with 5 total volume traded and no volume change in a week or two.

Thoughts?

Thanks again!

[WaterlessPiano2]

Cool, lets change cron job to be daily, and let add a logic on it to not fetch again if it has already fetched in the last 24 hours. Because currently, every time I run the cron job its fetching the data and inserting it yo supabase.

Yea true, it hasnt ocurred to me that the names are also unique, so yea we can use them as unique ids for now.

Yea i understand y u wanma drop old data, but i thought ur vision was to have historical data about projects no? if thats still the case then, then ithink we can look in to downloading the data or saving it to somewhere with cheaper storage option like AWS S3 as a compressed file, so if we ever need to do something with this data in the future, its not lost.

Yea İ definetly agree that we dont need to track dead projects. Lets not just track the SMB derivetives then. I prefer just tracking the top projects. I think once this project is inna good place, u can get some of the users to pay for it, and u should be able to pay for supabase.

On Thu, 20 Jan 2022, 00:26 David Reider, @.***> wrote:

Hi @WaterlessPiano2 https://github.com/WaterlessPiano2 !

We can definitely swap it to fetch data daily for now instead of hourly, that would help with the increasing database size (once we get the api.js running consistently).

Additionally, we can also change it to use the 'Business' derivatives, we would just need to adjust the logic for that. Getting the images wasn't super important - it was more of a 'want' than a need, but I do like the idea of a secondary table for that.

I'd have to check the JSON of the magic eden API to see if the projects have a unique ID to them, but I also think the names are unique so it may just use that (I don't think there can be two "Solana Monkey Business" projects).

In the end, we would probably still want to drop old data (or I'd have to purchase a bigger storage plan, but without a source of income on the project I am not sure I would be interested in spending a ton on a better plan currently) and we would also probably still want to only track projects that are gaining traction or are already decently sized. Probably no point in tracking a project with 5 total volume traded and no volume change in a week or two.

Thoughts?

Thanks again!

— Reply to this email directly, view it on GitHub https://github.com/DavidReider/MagicEdenFloorTrack/issues/11#issuecomment-1016993808, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHXPGNH6JHIED66MADXEOMTUW5JDFANCNFSM5MBWPQ6Q . You are receiving this because you were mentioned.Message ID: @.***>

[DavidReider]

I changed the CRON job to run daily at 03:00 (once we get it up on a server and running automatically). I also added some logic to only fetch projects with over 10k volume. This should help cut down on projects being tracked. I am not sure there is a need to set up logic to not fetch again if it has fetched in the last day because once it is running on the cron it should only fetch once daily instead of needing to run it manually ever.

My vision was to have some historic data to show for a specific project, but if necessary we can alter that plan :)

We can always adjust the logic as to what we want to track and what sort of limits we set for if a project will be tracked!