ghost
commented
1 year ago Translating things to SQL would not be easy especially for performance critical applications.
Ive heard it takes a lot of experience to build a high performance large SQL database.
Memcached along with memcachedb (https://github.com/LMDB/memcachedb/ ) also seems like nice alternatives.
Support for embedded databases like Berkley DB or even SQL lite seems more doable.
True base has posix like text files, may be a traditional telnet interface seems to be needed too :)
Just documenting some thoughts on how you'd scale TrueBase to 1.5M lines.
1) Folder splits: You'd probably want to shard the directory to ensure good performance with programs like Finder and Git that sometimes struggle with lots of files in one folder.
2) Server split: The current TrueBase implementation is an embedded server that starts and re-stops with the application. Assuming this thing continues to grow, at some point this will be too slow (to reread all files and rebuild indexes on startup), and it would be necessary to have an option to run a traditional server/client model.
3) Server rewrite: In the short-term I think we can just make the current thing faster and that will keep us moving. In the medium-term I think we can just use the existing code and adjust it to have 2 separate processes: one for the web server and one for the TrueBase server. In the long run (again, assuming TrueBase turns out to be a worthwhile idea), it may make sense to build a standalone TrueBase server in Rust, C, or GoLang. That would mean implementing Grammar + TQL in that language as well. Or perhaps you could build a layer that takes an on disk TrueBase, converts it accurately to a SQL structure, and rewrites TQL queries to SQL.