Add EDN Support

[adithyabsk]

Addresses the discussion in #64

Fixes #63

Make EDN support optional and other cleanups

• scrapping.py
  • Add ROAM_FORMATS const to define processable roam formats
  • refactor scrap to only require a tmp_dir and a list of formats to process
• fs.py
  • Simply formatting functions to a single save_files method that handles all of the different file types
  • rename unzip_markdown_archive --> unzip_archive
  • These two changes simplify unzip_and_save_archive which calls both of those two methods
  • Add optional jet pretty-printing and update --formats help string
• __main__.py
  • Add formats arg which allows the user to specify which formats they would like to save their files to
  • Remove skip fetch since it is now redundant
  • simply call with just --formats formatted
  • Simplify main execution to extend to more Roam formats if added.

[tylerwince]

Is this different than #64 ?

[adithyabsk]

@tylerwince No, it builds on #64

[adithyabsk]

@MatthieuBizien could you give this a once over?

[rushikb]

apologies for the noob q— but has this already been merged into the main branch? if so, do we need to do anything to configure our existing exports to include EDN? @MatthieuBizien @adithyabsk

[adithyabsk]

@rushikb no worries mate! I'm still waiting on the go-ahead from @MatthieuBizien, I don't have access to merge it in myself. Looks like he's been busy, if you want to feature right now, you can use my fork in the mean time and pip install from my repo.

[pcuellar]

@rushikb no worries mate! I'm still waiting on the go-ahead from @MatthieuBizien, I don't have access to merge it in myself. Looks like he's been busy, if you want to feature right now, you can use my fork in the mean time and pip install from my repo.

@adithyabsk Are you actively using this merge to backup your Roam db? If so, do you have any documentation on how the process differs from current workflow?

[cansar]

It would be fantastic to have this or something similar merged to get EDN supported in some format since the JSON export leaves out a lot of state.

[kerim]

Is EDN support included now? If so, how do I turn it on?

[rushikb]

Hi @MatthieuBizien - just wanted to flag this really quickly. Any chance you could take a look at the edn fix above and merge?

[adithyabsk]

@rushikb @pcuellar @cansar @kerim

I just updated my fork with a version of this PR that includes the latest changes. Feel free to use it until we can get a review from @MatthieuBizien.

Here's the GH Action I'm using:

name: "Roam Research Backup"

on:
  push:
    branches:
      - master
  schedule:
    -   cron: "5 */8 * * *"  # At minute 5 past every 8th hour.

jobs:
  backup:
    runs-on: macos-latest
    name: Backup
    timeout-minutes: 15
    steps:
      -   uses: actions/checkout@v2
      -   name: Setup Python 3.8
          uses: actions/setup-python@v1
          with:
            python-version: 3.8

      -   name: Setup Dependencies
          run: |
            brew install borkdude/brew/jet
            pip install git+https://github.com/adithyabsk/roam-to-git.git@master
      -   name: Run backup
          run: |
            roam-to-git --skip-git -f json markdown formatted edn -- .
          env:
            ROAMRESEARCH_USER: ${{ secrets.ROAMRESEARCH_USER }}
            ROAMRESEARCH_PASSWORD: ${{ secrets.ROAMRESEARCH_PASSWORD }}
            ROAMRESEARCH_DATABASE: ${{ secrets.ROAMRESEARCH_DATABASE }}

      -   name: Commit Changes
          uses: stefanzweifel/git-auto-commit-action@v4
          with:
            commit_message: Automated snapshot
            commit_user_name: GitHub Actions
            commit_user_email: actions@github.com
            commit_author: GitHub Actions <actions@github.com>

[everruler12]

Since I don't know enough about Python to make pull requests here, I wrote a Node backup solution from scratch, and just added EDN support https://github.com/everruler12/roam2github

[kerim]

Thanks for this @everruler12!

[pmbauer]

For what it's worth, roam-to-git is working again for me. @adithyabsk 's fork in particular has the advantage of formatted edn which is quite fast with jet.

JSON and markdown-based backups are fundamentally broken.

@adithyabsk this is the workflow I'm using, uses ubuntu instead of mac os, downloads jet directly and skips brew, makes the dependency step only take a few seconds.

(this uses my fork of roam-to-git which is functionally identical to the adithyabsk fork but I don't recommend you use either directly; make your own fork for security reasons)

name: "Roam Research Backup"

on:
  push:
    branches:
      - master
  schedule:
    -   cron: "0 */2 * * *"

jobs:
  backup:
    runs-on: ubuntu-latest
    name: Backup
    timeout-minutes: 15
    steps:
      -   uses: actions/checkout@v2
      -   name: Set up Python 3.8
          uses: actions/setup-python@v1
          with:
            python-version: 3.8

      -   name: Setup dependencies
          run: |
            wget -qO /tmp/jet.zip https://github.com/borkdude/jet/releases/download/v0.0.13/jet-0.0.13-linux-amd64.zip
            unzip /tmp/jet.zip -d /tmp
            sudo mv /tmp/jet /usr/local/bin/jet
            pip install git+https://github.com/pmbauer/roam-to-git.git
      -   name: Run backup
          run: |
            roam-to-git --skip-git -f markdown formatted edn -- .
          env:
            ROAMRESEARCH_USER: ${{ secrets.ROAMRESEARCH_USER }}
            ROAMRESEARCH_PASSWORD: ${{ secrets.ROAMRESEARCH_PASSWORD }}
            ROAMRESEARCH_DATABASE: ${{ secrets.ROAMRESEARCH_DATABASE }}

      -   name: Commit Changes
          uses: stefanzweifel/git-auto-commit-action@v4
          with:
            commit_message: Automated snapshot
            commit_user_name: GitHub Actions
            commit_user_email: actions@github.com
            commit_author: GitHub Actions <actions@github.com>

[kerim]

Is "formatted" EDN different from EDN? Which does @everruler12 provide?

[ErikPlachta]

What is the best solution to re import into my database if necessary?

[everruler12]

That's explained in my README

[pmbauer]

@kerim formatted as in pretty-printed. The EDN returned by roam is all on one line and not diffable so doesn't work well if you are trying to version your backups with git. I haven't tried @everruler12's solution so I can't speak to its performance, but @adithyabsk fork uses https://github.com/borkdude/jet, a graal-vm native binary using https://github.com/brandonbloom/fipp for pretty printing and it's decently fast on my largish graph, < 1min to format.

@ErikPlachta EDN is the only option for restoring a full backup. JSON and Markdown are both lossy (block-level backlinks, calculations, versions, all lost) and Markdown in particular has a 10-page import limit.

[adithyabsk]

Markdown in particular has a 10-page import limit

You're right! But you can in fact get around that since that limit is only client side

[pmbauer]

@adithyabsk yep ;) It's hacky. When I wrote that note on the forums it was still working, but I've since run into practical limits around 100+ pages so I don't recommend it. There is still no good easy solution to share sub-graphs for publishing via the json and markdown exports unfortunately.

[adithyabsk]

@pmbauer Oh hahahaha, I didn't realize I was citing you. I was on mobile and just pulled the link from another project I worked on:

https://github.com/adithyabsk/keep2roam

[everruler12]

@pmbauer Thanks for the explanation behind the fork! My solution is compiled with Lumo, a ClojureScript environment (so it uses clojure.pprint). The consensus seems to be that's slower than fipp. It does take a couple minutes to format my graph's EDN. I'm brand new to anything Clojure-related, so I'll look into incorporating fipp next

[MatthieuBizien]

Hi @adithyabsk ! I finally had the time to review your PR, last months have been rough at work. It looks good to me. Thanks for your patience :)

[adithyabsk]

@MatthieuBizien absolutely no worries, haha we're all volunteers here : )

[adithyabsk]

@MatthieuBizien btw, I'm still up to help get this repo on pypi and clean up build and release process if you need it. I'll just need repo owner access