search

Schwad / PorkCasts-Montana

http://www.porkcast.com/
0 stars 0 forks source link

Fix Dead Database #17

Closed Schwad closed 8 years ago

Schwad commented 8 years ago

4,000,000 unnecessary objects in the database, which effectively broke porkcast. Homeless checks are most of it. surgically kill and clean them out in console. This should bring PC back online.

Schwad commented 8 years ago

There is probably a faster way. But since I killed the SOURCE of this problem this should be a one-time-only kill. Cycling through all the checks I check if there is a Query that matches user_id on the check. If there isn't then it's a homeless check and I destroy it. This should clear out far and away most of the 4,000,000+ unneeded objects. Unfortunately script seems like it will take a few hours to execute. Will try to tidy up some other things while it is running. Hopefully this brings us about ready for porkcast to be live again; although long term we will have to face database limitation issues creatively.

screen shot 2015-10-14 at 12 39 07 pm
Schwad commented 8 years ago

It's still only, like, 20% done.

Schwad commented 8 years ago

Well, today may be the day. Over halfway deleted.

Schwad commented 8 years ago

Seriously. There has to be a better way. I've gotten it down from 4,600,000 to 16,000 .. . Still far above the 10,000 I'm allowed. Will have to work out a solution that still elegantly stores the data, serves it up, doesn't hit the API every time the page opens, but can handle a lot more user queries and access.

structure:

user --> has_many --> queries query --> has_many --> checks query --> has_many --> creditcards

So if I query nick schwaderer and he has 150 checks in his history, I have just kicked off 150 objects in my database. Maybe serialize everything into one object? It will make nightly comparison scripts trickier but might be doable.

Schwad commented 8 years ago

I am half tempted to close this issue as the faulty parts of the database are dead, but I will leave it open until our bigger issue of trying to operate this application on a 10,000 row heroku hobby-dev acct is resolved.

Schwad commented 8 years ago
screen shot 2015-10-19 at 11 28 44 pm
Schwad commented 8 years ago

Alright, short of breaking everything and going weird/abstract I think we have done a good job reducing it to 15,000 rows. Considering that an upgrade to 10,000,000 rows is only $9/month, I like our current setup. Think about it, as broken as it was (literally 400 times more inefficient than current settings) it still would not have broken the threshold. Under our current setup, I project us being able to comfortably handle 5,000 active users under that upgrade, which in Montana is unlikely. (Montana Policy Institute, a formerly powerful and influential site that updated daily with my work, only ever saw 100-120 unique visitors a day avg)

So right now I will leave this issue until I have the cashola lined up to handle that cost. Unfortunately my little email including a fundraising thing did not raise any money so this will have to be dealt with later.

Info on pricing here: https://www.heroku.com/pricing

Schwad commented 8 years ago

OFFICIALLY MIGRATED OVER TO 10,000,000 DATABASE HALLELUJAH. Very scary stuff.