To Do list for December (2018)

[justinclift]

DBHub.io

• [ ] Update and tweak email notifications, as per #112
• [ ] Include branch name in JSON listings #108 9caf4346c2e9174067b5be66ad41eaacc09b86c1 - * in progress *
  • Waiting for feedback from @MKleusberg
• [ ] Consider adding PRAGMA cell_size_check=ON and PRAGMA integrity_check to our uploaded database check, as per https://news.ycombinator.com/item?id=15612447
  • In progress.
  • [x] integrity_check has been added to the uploaded database sanity check code 6a26711e9616836dafd088aa8910c04a7c6abf2f
  • [ ] cell_size_check=ON should probably be added to the general SQLite querying code
• [ ] Test the webUI on iOS
  • [ ] Safari
  • [ ] Firefox
• [ ] Is there a way to allow for anonymous readers of public databases via DB4S?
  • So people can download public databases without needing to install a cert?
  • One approach would be to bundle a default "anonymous@dbhub.io" certificate with DB4S, which allows for read only usage but not uploading. That would probably not be too hard to get done.
• [ ] When people download a database or CSV file, send them the compressed (.zip) version
  • Or at least add an option to the download button dropdown, making it a chooseable option
• [ ] Find out why DB4S uploaded databases weren't invalidating memcache entries
• [ ] Finish the "Expanded SQL" PR for https://github.com/gwenn/gosqlite/issues/10#issuecomment-297830554
  • Initial PR fails on Travis: https://github.com/gwenn/gosqlite/pull/11
• [ ] Check if the contributors count still isn't updated when deleting commits/rewriting history. It should, as the # of contributors can go down when this occurs
• [ ] Add a "Create new branch" button to the branches page #71
• [ ] Add a "Create new tag" button to the tags page #71
• [ ] Add a "Create new release" button to the releases page #71
• [ ] Initial CSV upload implementation. As per #28.
• [ ] Create the front page
  • [ ] Data sets for exploring
  • [ ] Header bar
  • [ ] Features
  • [ ] Videos
  • [ ] Docs
  • [ ] API
• [ ] Delete releases needs an 'are you sure?' #71
• [ ] When renaming a database, add a check to make sure a database of that name doesn't already exist for the user
• [ ] Add sorting options to the database lists on the profile and user pages #77
  • Sort by name and sort by last modified
  • Drop down selector maybe?
• [ ] Figure out a good way to guide users in creating a new commit, as it's not immediately obvious atm #71
• [ ] Add auto-restart for the webUI launch process so Golang crashes don't kill the site #73
• [ ] Add a bunch of example databases to the new db4s-beta server
  • Some NICVA databases added.
  • There was a bug in Auth0 stopping the upload of databases from data.gov.uk.
  • Bug # 26204 filed with Auth0.
  • Fixed in the new v10.20.0 release of lock.
• [ ] Run EXPLAIN ANALYZE on all of our queries in postgresql.go, making sure the indices and similar are reasonable
  • Might be useful while doing this: https://github.com/dalibo/hypopg
  • Automatic indexing: https://github.com/ankane/dexter
  • Ugh. Ruby. Looking at the code, it's pretty simple. Likely not difficult to copy the concept to a Go version
• [ ] Make sure all stored data structures are as expected (eg commit->tree->entries, and commit sha256 structure internals), just to be on the safe side
  • Make sure the general structure is ok
  • It might also be useful to create a quick and dirty CLI tool which reads + validates the json structures in database entries (in bulk)
• [ ] Significant quantity of testing :smile:
• [ ] Allow Auth0 database users to change their password, as per #44
• [ ] Add further DB4S end point pieces
  • Much of this stuff probably won't be used right away, but having it in place gives us options to enhance the Remote pane in DB4S as time/interest permits
  • [ ] Query for user info
  • [ ] Full name
  • [ ] Date joined
  • [ ] Gravatar image?
  • [ ] Delete remote database
  • [ ] Delete latest commit (for a branch), tag, release, branch
  • [ ] Create/update tag, branch, release
• [ ] The headings on the front page ("Most downloaded" etc) should be clickable, and go to a full page showing many results (top 250 or something maybe?)
• [ ] API server and dio
  • We'll need to figure out the authentication approach for the API server and dio too
  • Certs or API key? API key is probably lowest overhead, as certs would add kB's and latency to every request
  • On the other hand, unless we're planning on supporting plain http we're going to be using certs to some extent anyway. And DB4S requires the use of client side certs.
  • Looking around online (quickly), it seems like HTTPS-only is the direction for API's these days. So, the session overhead of HTTPS is probably "just how it is" these days.
    • Example reference info (there are likely better sources): https://https.cio.gov/apis
• [ ] Add the new Linux Foundation "Community Data Licences" to the default licence list
  • https://cdla.io
  • They seem to effectively do the same things as the CC ones, so not sure how much use they'll be in the real world. Shouldn't hurt to add them as options though anyway.
  • Looking at this again (on 2017-12-14), these two new CDLA licenses seem to have gotten no traction whatsoever. From anyone.
  • They added a mailing list "cdla-discuss" after their announcement. It's not public, but can be subscribed to easily. After subscribing... that mailing list has a total of 43 members (including me) and there are 0 messages in the group. Really seems like a dead effort, not worth wasting time on.

DB4S

• [x] Look into https://github.com/sqlitebrowser/sqlitebrowser/issues/1192
• [ ] Does https://www.sqlite.org/src/info/04925dee41a21ffc affect us?
• [ ] Look into enabling SQLITE_ENABLE_STAT4 for the SQLite and SQLCipher compilation on OSX and Windows
  • https://github.com/sqlitebrowser/sqlitebrowser/issues/1161
• [ ] Look into the OSX stuttering problem: https://github.com/sqlitebrowser/sqlitebrowser/issues/1134 - *in progress*
  • [x] Try out the 3rd May nightly build - just prior to the QScintilla update - to see if the problem is present in that
  • The problem does seem to arrive with our update to QScintilla 2.10.0. Tried a few potential source changes to see if a smoking gun could be found, but nothing so far. 😦
• [ ] Create a .sql script to run after the nightly builds, which tests them to ensure the JSON1 and FTS extensions are working
  • The simple test in the Homebrew SQLCipher formula would probably be a decent start
• [x] Test out Hugo as a potential blog platform (aka sqlitebrowser.org/blog)
• [ ] Testing DB4S fixes
  • [ ] https://github.com/sqlitebrowser/sqlitebrowser/issues/583
• [ ] File issues on the upstream projects causing the DB4S compilation warnings https://github.com/sqlitebrowser/sqlitebrowser/issues/1042#issuecomment-311657316
• [ ] Update the DB4S installation docs for Linux
  • [ ] Debian/Ubuntu
  • Related:
  • https://github.com/sqlitebrowser/sqlitebrowser/issues/832#issuecomment-271159422
  • https://github.com/sqlitebrowser/sqlitebrowser/issues/994

[justinclift]

Closing this for now, as it was growing out of hand. I'll pull individual pieces out of it as needed.