Tie the data formats together

[michielbdejong]

There is quite some code in run.js and chart.html and it uses similar but incompatible data formats. Should tie those together! Would Cambria be useful for that? Or maybe bring it all back to the HalfTrade class and improve that?

[michielbdejong]

Following discussion at Federated Bookkeeping Weekly Call, I realised that the format/ontology in which personal finance data is stored in a user's pod, needs not be standardised at all. We can use a 'data lake approach' and store the raw messages/imports along with metadata about them, but not transforming the data before storing it to the user's pod.

The code then transforms the data on the fly, to produce the desired view to the end-user.

At most, there may be an intermediate data format to cache, but this could then be much closer to the view than you would normally do for a data(base) schema in a stand-alone software system.

[michielbdejong]

data sources that go into a view:

• bank statement csv / mt940 documents
• time sheets (json?)
• expenses (json?)
• projects (json?)
• invoices (peppol-bis?)
• depreciation hints
• budget hints
• query (e.g. months you want to report, type of graph / output format you want)

code modules:

1. parsers (for bank statements but also for expenses and timesheet data)
2. merging and matching
3. implied transactions
4. budgets
5. depreciation
6. work
7. views

[michielbdejong]

The parsers generate Entrys and information about completeness of the view for some dates Merging and Matching brings data from all sources together into one big list, and filters out the duplicates. Implied transactions, budgets, and depreciation add three types of additional transactions:

• a payment at a shop implies a purchase and delivery of value from that shop to you, so your assets increase
• depreciation determines how much you consume from your assets
• one of your budgets is decreased for each type of consumption. the budget entry is linked to the depreciation entry which is linked to the purchase entry which is linked to the payment entry which has the additional details with hints for which budget to automatically assign
• the work module deals with expenses, timesheets, invoices, and projects
• views list and display the data for different kinds of insights

[michielbdejong]

To what extent is this bookkeeping system federated? It's nice that expenses and time sheets can be federated between organisation and employee. This is probably also possible for virtual organisations. invoices can be sent with e-invoicing we could experiment with federated timesheets

[michielbdejong]

should I continue this work in PHP?

[michielbdejong]

Ha, thanks to @paul-french we discovered https://plaintextaccounting.org/ and https://hledger.org/ - will try out if we can switch to that!

[michielbdejong]

Wow, this is very valuable already: https://github.com/simonmichael/hledger/tree/master/examples/csv

[michielbdejong]

And more importers listed on https://plaintextaccounting.org/#data-importconversion