Closed adithyabsk closed 3 years ago
Is this different than #64 ?
@tylerwince No, it builds on #64
@MatthieuBizien could you give this a once over?
apologies for the noob q— but has this already been merged into the main branch? if so, do we need to do anything to configure our existing exports to include EDN? @MatthieuBizien @adithyabsk
@rushikb no worries mate! I'm still waiting on the go-ahead from @MatthieuBizien, I don't have access to merge it in myself. Looks like he's been busy, if you want to feature right now, you can use my fork in the mean time and pip install from my repo.
@rushikb no worries mate! I'm still waiting on the go-ahead from @MatthieuBizien, I don't have access to merge it in myself. Looks like he's been busy, if you want to feature right now, you can use my fork in the mean time and pip install from my repo.
@adithyabsk Are you actively using this merge to backup your Roam db? If so, do you have any documentation on how the process differs from current workflow?
It would be fantastic to have this or something similar merged to get EDN supported in some format since the JSON export leaves out a lot of state.
Is EDN support included now? If so, how do I turn it on?
Hi @MatthieuBizien - just wanted to flag this really quickly. Any chance you could take a look at the edn fix above and merge?
@rushikb @pcuellar @cansar @kerim
I just updated my fork with a version of this PR that includes the latest changes. Feel free to use it until we can get a review from @MatthieuBizien.
Here's the GH Action I'm using:
name: "Roam Research Backup"
on:
push:
branches:
- master
schedule:
- cron: "5 */8 * * *" # At minute 5 past every 8th hour.
jobs:
backup:
runs-on: macos-latest
name: Backup
timeout-minutes: 15
steps:
- uses: actions/checkout@v2
- name: Setup Python 3.8
uses: actions/setup-python@v1
with:
python-version: 3.8
- name: Setup Dependencies
run: |
brew install borkdude/brew/jet
pip install git+https://github.com/adithyabsk/roam-to-git.git@master
- name: Run backup
run: |
roam-to-git --skip-git -f json markdown formatted edn -- .
env:
ROAMRESEARCH_USER: ${{ secrets.ROAMRESEARCH_USER }}
ROAMRESEARCH_PASSWORD: ${{ secrets.ROAMRESEARCH_PASSWORD }}
ROAMRESEARCH_DATABASE: ${{ secrets.ROAMRESEARCH_DATABASE }}
- name: Commit Changes
uses: stefanzweifel/git-auto-commit-action@v4
with:
commit_message: Automated snapshot
commit_user_name: GitHub Actions
commit_user_email: actions@github.com
commit_author: GitHub Actions <actions@github.com>
Since I don't know enough about Python to make pull requests here, I wrote a Node backup solution from scratch, and just added EDN support https://github.com/everruler12/roam2github
Thanks for this @everruler12!
For what it's worth, roam-to-git
is working again for me. @adithyabsk 's fork in particular has the advantage of formatted edn which is quite fast with jet.
JSON and markdown-based backups are fundamentally broken.
@adithyabsk this is the workflow I'm using, uses ubuntu instead of mac os, downloads jet
directly and skips brew, makes the dependency step only take a few seconds.
(this uses my fork of roam-to-git
which is functionally identical to the adithyabsk
fork but I don't recommend you use either directly; make your own fork for security reasons)
name: "Roam Research Backup"
on:
push:
branches:
- master
schedule:
- cron: "0 */2 * * *"
jobs:
backup:
runs-on: ubuntu-latest
name: Backup
timeout-minutes: 15
steps:
- uses: actions/checkout@v2
- name: Set up Python 3.8
uses: actions/setup-python@v1
with:
python-version: 3.8
- name: Setup dependencies
run: |
wget -qO /tmp/jet.zip https://github.com/borkdude/jet/releases/download/v0.0.13/jet-0.0.13-linux-amd64.zip
unzip /tmp/jet.zip -d /tmp
sudo mv /tmp/jet /usr/local/bin/jet
pip install git+https://github.com/pmbauer/roam-to-git.git
- name: Run backup
run: |
roam-to-git --skip-git -f markdown formatted edn -- .
env:
ROAMRESEARCH_USER: ${{ secrets.ROAMRESEARCH_USER }}
ROAMRESEARCH_PASSWORD: ${{ secrets.ROAMRESEARCH_PASSWORD }}
ROAMRESEARCH_DATABASE: ${{ secrets.ROAMRESEARCH_DATABASE }}
- name: Commit Changes
uses: stefanzweifel/git-auto-commit-action@v4
with:
commit_message: Automated snapshot
commit_user_name: GitHub Actions
commit_user_email: actions@github.com
commit_author: GitHub Actions <actions@github.com>
Is "formatted" EDN different from EDN? Which does @everruler12 provide?
What is the best solution to re import into my database if necessary?
That's explained in my README
@kerim formatted as in pretty-printed. The EDN returned by roam is all on one line and not diffable so doesn't work well if you are trying to version your backups with git. I haven't tried @everruler12's solution so I can't speak to its performance, but @adithyabsk fork uses https://github.com/borkdude/jet, a graal-vm native binary using https://github.com/brandonbloom/fipp for pretty printing and it's decently fast on my largish graph, < 1min to format.
@ErikPlachta EDN is the only option for restoring a full backup. JSON and Markdown are both lossy (block-level backlinks, calculations, versions, all lost) and Markdown in particular has a 10-page import limit.
Markdown in particular has a 10-page import limit
You're right! But you can in fact get around that since that limit is only client side
@adithyabsk yep ;) It's hacky. When I wrote that note on the forums it was still working, but I've since run into practical limits around 100+ pages so I don't recommend it. There is still no good easy solution to share sub-graphs for publishing via the json and markdown exports unfortunately.
@pmbauer Oh hahahaha, I didn't realize I was citing you. I was on mobile and just pulled the link from another project I worked on:
@pmbauer Thanks for the explanation behind the fork! My solution is compiled with Lumo, a ClojureScript environment (so it uses clojure.pprint
). The consensus seems to be that's slower than fipp. It does take a couple minutes to format my graph's EDN. I'm brand new to anything Clojure-related, so I'll look into incorporating fipp next
Hi @adithyabsk ! I finally had the time to review your PR, last months have been rough at work. It looks good to me. Thanks for your patience :)
@MatthieuBizien absolutely no worries, haha we're all volunteers here : )
@MatthieuBizien btw, I'm still up to help get this repo on pypi and clean up build and release process if you need it. I'll just need repo owner access
Addresses the discussion in #64
Fixes #63
Make EDN support optional and other cleanups
scrapping.py
ROAM_FORMATS
const to define processable roam formatsfs.py
save_files
method that handles all of the different file typesunzip_markdown_archive
-->unzip_archive
unzip_and_save_archive
which calls both of those two methodsjet
pretty-printing and update --formats help string__main__.py
--formats formatted