search

idno / known

A social publishing platform.
https://withknown.com/opensource
Other
1.05k stars 195 forks source link

Flat File Storage in Known #1631

Closed herbsmn closed 7 years ago

herbsmn commented 7 years ago

Geez. I'm paying an incredible amount in DBA taxes [1] when hacking on my self hosted VPS that is running Known. I'm sure a large amount of this is due to me being a bit of a n00b, but I've mostly found that Known works perfectly most all of the time, but database problems often add tens of hours of headaches every few months with the two Known websites I've been running.

Also, I'm bitter right now because my MongoDB database got hacked and now I can't access over 2 years worth of Known content. On my other Known site, mariaDB stopped working after I upgraded it and it took 3 months to finally get it working again. I am a hobbyist, I'll admit, and I don't work on my website all the time, but this prevented me from using Known, which I love to publish on, for a good while there. It sounds dumb that it took me so long, but it probably took me 20 hours to get mariadb running again after it entered a failed state on an upgrade. I would have rather spent that time hacking Known, but I should also disclose that part of the problem was that I was newly familiarizing myself with SystemD as well and dealing with that didn't come intuitively.

After taking a peek on their wiki, it seems like there are a number of people in the IndieWeb community that advocate against complex databases like MongoDB, MariaDB, PostgreSQL and MySQL.[2]

They suggest that complex databases work against a core principles [3] of the IndieWeb movement, the goal of longevity [4] of one's content and data (including permalinks).[5] They say we should try to build for the long web. [6]

I am not trying to bash on Known for only using complex databases, but I would like to start a discussion around whether Known could be modified to store main (primary) site content in flat files in a filesystem.[7] Perhaps needing a database is baked in to the program. If not, maybe someone will read the stuff in this thread and be inspired to try to add a plugin to add support for flat file storage.

Here's some links to people who aren't using complex database in the IndieWeb community.[8][9]

[1] https://indieweb.org/database-antipattern#DBA_tax [2] https://indieweb.org/database-antipattern [3] #9 https://indieweb.org/principles --> [4] https://indieweb.org/Longevity [5] https://indieweb.org/database-antipattern#How_is_this_related_to_the_indieweb.3F [6] http://adactio.com/articles/1522/ [7] https://indieweb.org/file-storage [8] https://indieweb.org/database-antipattern#Alternatives [9] http://indieweb.org/file-storage

Edit: I changed a lot of the text of this after posting it because I really disliked the tone after I re-read it.

herbsmn commented 7 years ago

I don't know, maybe the majority of my issues are related to not working hard enough to learn mariadb / MySQL.

Also, it seems as though postres users seem to not feel very burdened by their database software based on these reports: https://indieweb.org/Postgres

I've used it a little bit but haven't gotten into it much. I really appreciate that Known supports Postgres. Thanks for the hard work that you guys have done and specifically for adding Postgres support. I'm going to test out Known+Postgres for my next site and see how it goes. Also, I'm going to try to understand SQL better in general, since I'll likely need to as I install more web apps in the future.

mapkyca commented 7 years ago

Closing this, this sort of discussion should go to the discussion group https://groups.google.com/forum/#!forum/known-dev or #knownchat IRC channel.

But, in short there are a number of very good reasons why Known uses a (number of different options of) database for storage.

For those who feel strongly about avoiding database for backend storage, and are prepared to accept the consequences, I encourage you to look at the DataConcierge class hierarchy. It is entirely possible to write a plugin that maps the data queries to another mechanism/flat file store (although I imagine search will suffer unless some effort is taken to provide some sort of lookup mechanism), however I doubt this'll ever be something that is built out for the core project.