MarkdownDB: extracting links plus refactor

[olayway]

Closes datopian/markdowndb#12

Motivation

As we plan to replace Contentlayer with our MarkdownDB package, we need a solid base upon which we can build a full-fledged "content-layer" solution. As the code in its current state is pretty hard to reason about and manipulate and is rather weakly typed, the development of any new functionality on top of it would only make it harder and harder to engineer, adjust and test. This PR aims to make the development of new features easier and (because of stronger typing) less error-prone.

Note, there is still quite a lot of room for improvement - TODOs comments were left in the code but will probably be resolved after further discussions on implementation design.

Changes

• markdowndb.ts cleaned up and refactored. Most notably:
  • tables creation (and deletion) logic moved out to schema.ts and bound with relevant classes corresponding to tables (see below)
  • wrapped raw batch inserts on knex db with class methods to prevent SQL errors in the first place, instead of debugging them after they occur
  • indexFolder bound with MarkdownDB class as a method
  • removed duplication of db initialization -> there is only one init method on MarkdownDB now
  • replaced ambiguous query method with getFiles
  • commented out querying by folder (?) for now as I'm not sure if it's even needed, at least not in the current form
  • added getFileByUrl and getFileById method to retrieve single file file from the db (e.g. for finding its backlinks)
  • added getLinks method which supports querying back- or forward links
  • removed types.ts classes in schema.ts can also by used as types. Other types like e.g. DatabaseQuery moved directly to method signature, as they are striclty bound with it and it's not needed to have it in a separate file (there is no other use case for it)
  • cleaned up and extended tests of markdowndb lib and split into smaller unit tests grouped by tested functionalities
• schema.ts created with four classes: File, Link, Tag and FileTag
  • each class is now a representation of a table, i.e. files, links, tags, file_tags and so it describes the fields (columns) existent on each table and has methods for creating a table, deleting it and batch-inserting data to it
  • it is a single source of truth about tables and data types stored in them,
• utils folder created with recursiveWalkDir.ts, parseFile.ts and extractWikiLinks.ts each with its own, separate test suite
• stronger typing

[changeset-bot[bot]]

⚠️ No Changeset found

Latest commit: e4fefdcc280ab90d1534f5931d918dc6df9999d9

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[olayway]

My only concern is that we would need some sort of query by folder functionality to replace our current implementation of markdowndb on datahub-next, because otherwise we would have to modify all the files so that those would have the correct filetype (if I understood it right).

Got it. I'll bring this functionality back!

[netlify[bot]]

✅ Deploy Preview for spectacular-dragon-c1015c ready!

Name	Link
🔨 Latest commit	e4fefdcc280ab90d1534f5931d918dc6df9999d9
🔍 Latest deploy log	https://app.netlify.com/sites/spectacular-dragon-c1015c/deploys/642c28e0d68c1d0008636d75
😎 Deploy Preview	https://deploy-preview-472--spectacular-dragon-c1015c.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.