This script helps you backup your RoamResearch graphs!
This script automatically
V.02:
See it in action!. This repo is updated using roam-to-git.
🚀🚀 NEW 🚀🚀 : The Unofficial backup of the official RoamResearch Help Database
Note: Erik Newhard's guide shows an easy way of setting up GitHub Actions without using the CLI.
With gh: gh repo create notes
(yes, it's private)
Or manually
Due to the limitations of OAuth and complexities with tokens, we are unable to snapshot accounts that are set up with the Login with Google option as of now.
To set up backup in this case, you will need to create(not exactly) a native account from your old Google Account, which is as simple as using the reset password link found in Roam.
Once you've reset your password, use the following steps to finish setting up your backup!
Add 3 (separate) secrets where the names are
ROAMRESEARCH_USER
ROAMRESEARCH_PASSWORD
ROAMRESEARCH_DATABASE
Refer to env.template for more information
when inserting the information, there is no need for quotations or assignments
cd notes
mkdir -p .github/workflows/
curl https://raw.githubusercontent.com/MatthieuBizien/roam-to-git-demo/master/.github/workflows/main.yml > \
.github/workflows/main.yml
git add .github/workflows/main.yml
git commit -m "Add github/workflows/main.yml"
git push --set-upstream origin master
If the backup does not automatically start, try pushing to the repository again
This section is based on this article on GitLab blog: https://about.gitlab.com/blog/2017/11/02/automating-boring-git-operations-gitlab-ci/
Create a project for your notes. We will refer to it as
YOUR_USER/YOUR_PROJECT
.
Generate a new key pair that will be used by the CI job to push the new commits.
$ ssh-keygen -f gitlab-ci-commit
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in gitlab-ci-commit-test
Your public key has been saved in gitlab-ci-commit-test.pub
The key fingerprint is:
SHA256:HoQUcbUPJU2Ur78EineqA6IVljk8ZD9XIxiGFUrBues agentydragon@pop-os
The key's randomart image is:
+---[RSA 3072]----+
| .o=O*..o++. |
| .*+.o. o+o |
| +.=. .oo. . |
| X o.. o . |
| . = oS o. |
| + .. o ... |
| + . .o o ... |
| . E .. o .. |
| .o. .. |
+----[SHA256]-----+
DO NOT commit the private key (gitlab-ci-commit
).
This step allows the CI job to push when identified by the public key.
Go to Project Settings → Repository (https://gitlab.com/YOUR_USER/YOUR_PROJECT/-/settings/repository
) → Deploy Keys.
Paste the content of the public key file (gitlab-ci-commit-test.pub
), and
enable write access for the public key.
Click "Add Key".
This step gives the CI job the private key so it can authenticate against GitLab.
Go to Project Settings → Pipelines (https://gitlab.com/YOUR_USER/YOUR_PROJECT/-/settings/ci_cd
) → Variables.
Click "Add variable", with name GIT_SSH_PRIV_KEY
, and paste in the content
of the private key file (gitlab-ci-commit
). You probably want to mark
"Protect". You might want to look up GitLab docs on protected branches.
Click "Add variable".
Also add the following variables with appropriate values:
ROAMRESEARCH_USER
ROAMRESEARCH_PASSWORD
ROAMRESEARCH_DATABASE
gitlab_known_hosts
fileIn your repo, create and commit a gitlab_known_hosts
file containing the
needed SSH known_hosts
entry/entries for the GitLab instance. It will be used
by the CI job to check that it's talking to the right server.
This should work as of 2021-01-04:
# Generated by @agentydragon at 2021-01-04 at by `git fetch`ing a GitLab repo
# with an empty ~/.ssh/known_hosts file.
|1|zIQlCRxv+s9xhVCAfGL2nvaZqdY=|jbPpD9GNaS/9Z4iJzE9gw2XCo20= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=
|1|uj60xYhsW2vAM8BpQ+xZz51ZarQ=|BNIJlvu4rNcmrxd60fkqpChrf9A= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=
.gitlab-ci.yml
Create a .gitlab-ci.yml
file:
backup:
when: manual
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$GIT_SSH_PRIV_KEY")
- git config --global user.email "roam-to-git@example.org"
- git config --global user.name "roam-to-git automated backup"
- mkdir -p ~/.ssh
- cat gitlab_known_hosts >> ~/.ssh/known_hosts
# (Taken from: https://github.com/buildkite/docker-puppeteer/blob/master/Dockerfile)
# We install Chrome to get all the OS level dependencies, but Chrome itself
# is not actually used as it's packaged in the pyppeteer library.
# Alternatively, we could include the entire dep list ourselves
# (https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md#chrome-headless-doesnt-launch-on-unix)
# but that seems too easy to get out of date.
- apt-get install -y wget gnupg ca-certificates
- wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
- echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
- apt-get update
- apt-get install google-chrome-stable libxss1 python3-pip -y
- pip3 install git+https://github.com/MatthieuBizien/roam-to-git.git
# TODO(agentydragon): Create and publish Docker image with all deps already
# installed.
script:
# Need to clone the repo again over SSH, since by default GitLab clones
# the repo for CI over HTTPS, for which we cannot authenticate pushes via
# pubkey.
- git clone --depth=1 git@gitlab.com:YOUR_USER/YOUR_PROJECT
- cd YOUR_PROJECT
# --no-sandbox needed because Chrome refuses to run as root without it.
- roam-to-git --browser-arg=--no-sandbox .
Commit and push.
Go to https://gitlab.com/YOUR_USER/YOUR_PROJECT/-/jobs
,
press the Play button:
Shortly after you should see an automated commit added to master.
To run the script, say, every hour, go to the project's
CI / CD → Schedules (https://gitlab.com/YOUR_USER/YOUR_PROJECT/-/pipeline_schedules
).
Click "New schedule", fill out and submit (like this for every 15 minutes):
Note: if your file system is not case-sensitive, you will not backup notes that have the same name in different cases
With pipx (if you don't know pipx, you should look at it, it's wonderful!)
pipx install git+https://github.com/MatthieuBizien/roam-to-git.git
With gh: gh repo create notes
(yes, it's private)
Or manually
Then run git push --set-upstream origin master
curl https://raw.githubusercontent.com/MatthieuBizien/roam-to-git/master/env.template > notes/.env
vi .env
echo .env > notes/.gitignore; cd notes; git add .gitignore; git commit -m "Initial commit"
roam-to-git notes/
One-liner to run it with a cron every hour:
echo "0 * * * * '$(which roam-to-git)' '$(pwd)/notes'" | crontab -
NB: there are issues on Mac with a crontab.
Making roam-to-git
foolproof is hard, as it depends on Roam, on GitHub Action or the local environment,
on software not very stable (pyppeteer
we still love you 😉 )
and on the correct user configuration.
For debugging, please try the following:
ROAMRESEARCH_USER
, ROAMRESEARCH_PASSWORD
, ROAMRESEARCH_DATABASE
are correctly setuproam-to-git --debug
to check the authentication and download work[[links]]
#links
attribute::
[[ [[link 1]] [[link 2]] ]]
((link))
/
in file names{{TODO}}
to be compatible with GitHub markdown