search

MatthieuBizien / roam-to-git

Automatic RoamResearch backup to Git
MIT License
550 stars 133 forks source link

[macOS] command to backup fails as crontab only #43

Closed dgraziotin closed 3 years ago

dgraziotin commented 4 years ago

Describe the bug On macOS Catalina. The same command that I run from the console (and works) fails when run as crontab.

To Reproduce

  1. Run the backup command from the command line (I anonymized e-mail address and graph name):
/Users/daniel/.local/bin/roam-to-git /Users/daniel/BrainBackup
2020-08-17 10:40:18.015 | INFO     | roam_to_git.__main__:main:50 - Loading secrets from /Users/daniel/BrainBackup/.env
2020-08-17 10:40:18.020 | DEBUG    | roam_to_git.scrapping:download_rr_archive:55 - Creating browser
2020-08-17 10:40:18.400 | DEBUG    | roam_to_git.scrapping:download_rr_archive:55 - Creating browser
2020-08-17 10:40:19.052 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:92 - Configure downloads to /var/folders/nh/jr3vkxsx14d3c4pss49krrcc0000gn/T/tmphp6ahdp0
2020-08-17 10:40:19.091 | DEBUG    | roam_to_git.scrapping:signin:183 - Opening signin page
2020-08-17 10:40:19.147 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:92 - Configure downloads to /var/folders/nh/jr3vkxsx14d3c4pss49krrcc0000gn/T/tmptfrz3qf0
2020-08-17 10:40:19.186 | DEBUG    | roam_to_git.scrapping:signin:183 - Opening signin page
2020-08-17 10:40:23.731 | DEBUG    | roam_to_git.scrapping:signin:187 - Fill email 'myemail@myserver.ext'
2020-08-17 10:40:23.778 | DEBUG    | roam_to_git.scrapping:signin:187 - Fill email 'myemail@myserver.ext'
2020-08-17 10:40:28.943 | DEBUG    | roam_to_git.scrapping:signin:192 - Fill password
2020-08-17 10:40:28.994 | DEBUG    | roam_to_git.scrapping:signin:192 - Fill password
2020-08-17 10:40:30.103 | DEBUG    | roam_to_git.scrapping:signin:197 - Click on sign-in
2020-08-17 10:40:30.154 | DEBUG    | roam_to_git.scrapping:signin:197 - Click on sign-in
2020-08-17 10:40:32.363 | DEBUG    | roam_to_git.scrapping:go_to_database:207 - Load database from url 'https://roamresearch.com/#/app/myroam'
2020-08-17 10:40:32.407 | DEBUG    | roam_to_git.scrapping:go_to_database:207 - Load database from url 'https://roamresearch.com/#/app/myroam'
2020-08-17 10:40:33.415 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:102 - Wait for interface to load
2020-08-17 10:40:33.438 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:102 - Wait for interface to load
2020-08-17 10:40:39.683 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:130 - Launch download popup
2020-08-17 10:40:39.705 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:130 - Launch download popup
2020-08-17 10:40:42.043 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:144 - Checking download type
2020-08-17 10:40:42.072 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:144 - Checking download type
2020-08-17 10:40:42.091 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:159 - Downloading output of type markdown
2020-08-17 10:40:42.116 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:148 - Changing output type to json
2020-08-17 10:40:42.433 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:164 - Wait download of markdown to /var/folders/nh/jr3vkxsx14d3c4pss49krrcc0000gn/T/tmphp6ahdp0
2020-08-17 10:40:43.435 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:174 - File /var/folders/nh/jr3vkxsx14d3c4pss49krrcc0000gn/T/tmphp6ahdp0/Roam-Export-1597653643328.zip found for markdown
2020-08-17 10:40:44.436 | DEBUG    | roam_to_git.scrapping:download_rr_archive:76 - Closing browser markdown
2020-08-17 10:40:44.489 | DEBUG    | roam_to_git.scrapping:download_rr_archive:78 - Closed browser markdown
2020-08-17 10:40:46.480 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:159 - Downloading output of type json
2020-08-17 10:40:46.816 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:164 - Wait download of json to /var/folders/nh/jr3vkxsx14d3c4pss49krrcc0000gn/T/tmptfrz3qf0
2020-08-17 10:40:47.818 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:174 - File /var/folders/nh/jr3vkxsx14d3c4pss49krrcc0000gn/T/tmptfrz3qf0/Roam-Export-1597653647235.zip found for json
2020-08-17 10:40:48.821 | DEBUG    | roam_to_git.scrapping:download_rr_archive:76 - Closing browser json
2020-08-17 10:40:48.867 | DEBUG    | roam_to_git.scrapping:download_rr_archive:78 - Closed browser json
2020-08-17 10:40:48.867 | DEBUG    | roam_to_git.scrapping:scrap:253 - Scrapping finished
2020-08-17 10:40:48.872 | DEBUG    | roam_to_git.fs:save_markdowns:50 - Saving markdown to /Users/daniel/BrainBackup/markdown
2020-08-17 10:40:48.885 | DEBUG    | roam_to_git.fs:unzip_and_save_json_archive:62 - Saving json to /Users/daniel/BrainBackup/json
2020-08-17 10:40:48.945 | DEBUG    | roam_to_git.fs:save_markdowns:50 - Saving markdown to /Users/daniel/BrainBackup/formatted
2020-08-17 10:40:49.056 | DEBUG    | roam_to_git.fs:commit_git_directory:79 - Committing git repository /Users/daniel/BrainBackup/.git
2020-08-17 10:40:49.141 | DEBUG    | roam_to_git.fs:push_git_repository:85 - Pushing to origin
  1. Add the same command as crontab (crontab -e)
0 * * * * /Users/daniel/.local/bin/roam-to-git /Users/daniel/BrainBackup

Expected behavior I would expect the same output and a successful backup.

What I get (I just take the latest one I received in my mailbox):

Subject: Cron <daniel@mymacbook> /Users/daniel/.local/bin/roam-to-git /Users/daniel/BrainBackup
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=daniel>
X-Cron-Env: <USER=daniel>
Date: Sun, 16 Aug 2020 22:00:01 +0200 (CEST)

2020-08-16 22:00:01.019 | INFO     | roam_to_git.__main__:main:50 - Loading secrets from /Users/daniel/BrainBackup/.env
2020-08-16 22:00:01.026 | DEBUG    | roam_to_git.scrapping:download_rr_archive:55 - Creating browser
2020-08-16 22:00:01.389 | DEBUG    | roam_to_git.scrapping:download_rr_archive:55 - Creating browser
2020-08-16 22:00:03.232 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:92 - Configure downloads to /tmp/tmpij4r1sjm
2020-08-16 22:00:03.272 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:92 - Configure downloads to /tmp/tmpcg12_9fi
2020-08-16 22:00:03.336 | DEBUG    | roam_to_git.scrapping:signin:183 - Opening signin page
2020-08-16 22:00:03.372 | DEBUG    | roam_to_git.scrapping:signin:183 - Opening signin page
2020-08-16 22:00:10.268 | DEBUG    | roam_to_git.scrapping:signin:187 - Fill email 'myemail@myserver.ext'
2020-08-16 22:00:10.572 | DEBUG    | roam_to_git.scrapping:signin:187 - Fill email 'myemail@myserver.ext'
2020-08-16 22:00:22.892 | DEBUG    | roam_to_git.scrapping:download_rr_archive:76 - Closing browser json
2020-08-16 22:00:22.903 | DEBUG    | roam_to_git.scrapping:download_rr_archive:76 - Closing browser markdown
2020-08-16 22:00:23.059 | DEBUG    | roam_to_git.scrapping:download_rr_archive:78 - Closed browser json
2020-08-16 22:00:23.060 | ERROR    | __main__:<module>:8 - An error has been caught in function '<module>', process 'MainProcess' (20024), thread 'MainThread' (4372716992):
Traceback (most recent call last):
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/execution_context.py", line 85, in evaluateHandle
    'userGesture': True,

pyppeteer.errors.NetworkError: Protocol error (Runtime.evaluate): Cannot find context with specified id

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
> File "/Users/daniel/.local/bin/roam-to-git", line 8, in <module>
    sys.exit(main())
    │   │    └ <function main at 0x7fb9100ba290>
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/__main__.py", line 76, in main
    scrap(markdown_zip_path, json_zip_path, config)
    │     │                  │              └ <roam_to_git.scrapping.Config object at 0x7fb928052b50>
    │     │                  └ PosixPath('/tmp/tmpcg12_9fi')
    │     └ PosixPath('/tmp/tmpij4r1sjm')
    └ <function scrap at 0x7fb9100ba0e0>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/scrapping.py", line 252, in scrap
    asyncio.get_event_loop().run_until_complete(asyncio.gather(*tasks))
    │       │                                   │       │       └ [<coroutine object download_rr_archive at 0x7fb9100ba560>, <coroutine object download_rr_archive at 0x7fb9100ba7a0>]
    │       │                                   │       └ <function gather at 0x7fb9507d2c20>
    │       │                                   └ <module 'asyncio' from '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/__init__.py'>
    │       └ <built-in function get_event_loop>
    └ <module 'asyncio' from '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/__init__.py'>
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
           │      └ <method 'result' of '_asyncio.Future' objects>
           └ <_GatheringFuture finished exception=NetworkError('Execution context was destroyed, most likely because of a navigation.')>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/scrapping.py", line 69, in download_rr_archive
    return await _download_rr_archive(document, output_type, output_directory, config)
                 │                    │         │            │                 └ <roam_to_git.scrapping.Config object at 0x7fb928052b50>
                 │                    │         │            └ PosixPath('/tmp/tmpcg12_9fi')
                 │                    │         └ 'json'
                 │                    └ <pyppeteer.page.Page object at 0x7fb950b1a5d0>
                 └ <function _download_rr_archive at 0x7fb93009c9e0>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/scrapping.py", line 97, in _download_rr_archive
    await signin(document, config, sleep_duration=config.sleep_duration)
          │      │         │                      │      └ 2.0
          │      │         │                      └ <roam_to_git.scrapping.Config object at 0x7fb928052b50>
          │      │         └ <roam_to_git.scrapping.Config object at 0x7fb928052b50>
          │      └ <pyppeteer.page.Page object at 0x7fb950b1a5d0>
          └ <function signin at 0x7fb93009c830>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/scrapping.py", line 188, in signin
    email_elem = await document.querySelector("input[name='email']")
                       │        └ <function Page.querySelector at 0x7fb9100a49e0>
                       └ <pyppeteer.page.Page object at 0x7fb950b1a5d0>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/page.py", line 371, in querySelector
    return await frame.querySelector(selector)
                 │     │             └ "input[name='email']"
                 │     └ <function Frame.querySelector at 0x7fb910063320>
                 └ <pyppeteer.frame_manager.Frame object at 0x7fb8f0010d90>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/frame_manager.py", line 316, in querySelector
    document = await self._document()
                     │    └ <function Frame._document at 0x7fb9100633b0>
                     └ <pyppeteer.frame_manager.Frame object at 0x7fb8f0010d90>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/frame_manager.py", line 326, in _document
    document = (await context.evaluateHandle('document')).asElement()
                      │       └ <function ExecutionContext.evaluateHandle at 0x7fb928071710>
                      └ <pyppeteer.execution_context.ExecutionContext object at 0x7fb8f00218d0>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/execution_context.py", line 88, in evaluateHandle
    _rewriteError(e)
    └ <function _rewriteError at 0x7fb92806b710>
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/execution_context.py", line 237, in _rewriteError
    raise type(error)(msg)
               │      └ 'Execution context was destroyed, most likely because of a navigation.'
               └ NetworkError('Protocol error (Runtime.evaluate): Cannot find context with specified id')

pyppeteer.errors.NetworkError: Execution context was destroyed, most likely because of a navigation.
Traceback (most recent call last):
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/execution_context.py", line 85, in evaluateHandle
    'userGesture': True,
pyppeteer.errors.NetworkError: Protocol error (Runtime.evaluate): Cannot find context with specified id

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/daniel/.local/bin/roam-to-git", line 8, in <module>
    sys.exit(main())
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/loguru/_logger.py", line 1149, in catch_wrapper
    return function(*args, **kwargs)
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/__main__.py", line 76, in main
    scrap(markdown_zip_path, json_zip_path, config)
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/scrapping.py", line 252, in scrap
    asyncio.get_event_loop().run_until_complete(asyncio.gather(*tasks))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/scrapping.py", line 69, in download_rr_archive
    return await _download_rr_archive(document, output_type, output_directory, config)
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/scrapping.py", line 97, in _download_rr_archive
    await signin(document, config, sleep_duration=config.sleep_duration)
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/roam_to_git/scrapping.py", line 188, in signin
    email_elem = await document.querySelector("input[name='email']")
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/page.py", line 371, in querySelector
    return await frame.querySelector(selector)
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/frame_manager.py", line 316, in querySelector
    document = await self._document()
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/frame_manager.py", line 326, in _document
    document = (await context.evaluateHandle('document')).asElement()
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/execution_context.py", line 88, in evaluateHandle
    _rewriteError(e)
  File "/Users/daniel/.local/pipx/venvs/roam-to-git/lib/python3.7/site-packages/pyppeteer/execution_context.py", line 237, in _rewriteError
    raise type(error)(msg)
pyppeteer.errors.NetworkError: Execution context was destroyed, most likely because of a navigation.
2020-08-16 22:00:23.086 | DEBUG    | roam_to_git.scrapping:_kill_child_process:215 - Terminate child process [psutil.Process(pid=20032, name='Chromium', status='zombie', started='22:00:01')]

Traceback Please use http://gist.github.com/ or similar, and report the last line here.

Run roam-to-git --debug notes/ and report what you get.

/Users/daniel/.local/bin/roam-to-git --debug /Users/daniel/BrainBackup
2020-08-17 10:51:15.974 | INFO     | roam_to_git.__main__:main:50 - Loading secrets from /Users/daniel/BrainBackup/.env
2020-08-17 10:51:15.977 | DEBUG    | roam_to_git.scrapping:download_rr_archive:55 - Creating browser
2020-08-17 10:51:17.654 | DEBUG    | roam_to_git.scrapping:signin:183 - Opening signin page
2020-08-17 10:51:22.112 | DEBUG    | roam_to_git.scrapping:signin:187 - Fill email 'myemail@myserver.ext'
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error (Runtime.releaseObject): Cannot find context with specified id')>
pyppeteer.errors.NetworkError: Protocol error (Runtime.releaseObject): Cannot find context with specified id
2020-08-17 10:51:27.398 | DEBUG    | roam_to_git.scrapping:signin:192 - Fill password
2020-08-17 10:51:28.572 | DEBUG    | roam_to_git.scrapping:signin:197 - Click on sign-in
2020-08-17 10:51:30.821 | DEBUG    | roam_to_git.scrapping:go_to_database:207 - Load database from url 'https://roamresearch.com/#/app/myroam'
2020-08-17 10:51:31.919 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:102 - Wait for interface to load
2020-08-17 10:51:38.225 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:130 - Launch download popup
2020-08-17 10:51:40.586 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:144 - Checking download type
2020-08-17 10:51:40.635 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:159 - Downloading output of type markdown
2020-08-17 10:51:40.970 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:164 - Wait download of markdown to /var/folders/nh/jr3vkxsx14d3c4pss49krrcc0000gn/T/tmpy8e_6hso
2020-08-17 10:51:40.970 | DEBUG    | roam_to_git.scrapping:download_rr_archive:55 - Creating browser
2020-08-17 10:51:42.647 | DEBUG    | roam_to_git.scrapping:signin:183 - Opening signin page
2020-08-17 10:51:46.838 | DEBUG    | roam_to_git.scrapping:signin:187 - Fill email 'myemail@myserver.ext'
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error (Runtime.releaseObject): Cannot find context with specified id')>
pyppeteer.errors.NetworkError: Protocol error (Runtime.releaseObject): Cannot find context with specified id
2020-08-17 10:51:52.303 | DEBUG    | roam_to_git.scrapping:signin:192 - Fill password
2020-08-17 10:51:53.458 | DEBUG    | roam_to_git.scrapping:signin:197 - Click on sign-in
2020-08-17 10:51:55.713 | DEBUG    | roam_to_git.scrapping:go_to_database:207 - Load database from url 'https://roamresearch.com/#/app/myroam'
2020-08-17 10:51:56.786 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:102 - Wait for interface to load
2020-08-17 10:52:03.071 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:130 - Launch download popup
2020-08-17 10:52:05.426 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:144 - Checking download type
2020-08-17 10:52:05.475 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:148 - Changing output type to json
2020-08-17 10:52:09.842 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:159 - Downloading output of type json
2020-08-17 10:52:10.183 | DEBUG    | roam_to_git.scrapping:_download_rr_archive:164 - Wait download of json to /var/folders/nh/jr3vkxsx14d3c4pss49krrcc0000gn/T/tmp77q3gvgy
2020-08-17 10:52:10.183 | WARNING  | roam_to_git.scrapping:scrap:249 - Exiting without updating the git repository, because we can't get the downloads with the option --debug
2020-08-17 10:52:10.183 | DEBUG    | roam_to_git.__main__:main:78 - waiting for the download...

Please complete the following information:

Additional context Add any other context about the problem here.

MatthieuBizien commented 4 years ago

Well I have the same error now. It uses to work. No idea where it's from, but Mac crontab is weird.

sinoth commented 4 years ago

I'm getting the same error running the script through a github action. The error is "Execution context was destroyed, most likely because of a navigation".

I'm only running once a day so I don't know how recurrent the problem is. My first error was last night.

cumtnp commented 4 years ago

@MatthieuBizien try my solution：https://github.com/MatthieuBizien/roam-to-git/pull/58 1，Wait for this element input[name='email'] on the webpage to be rendered。 2，Adjust the operating time slightly

MatthieuBizien commented 3 years ago

The backup errors should be fixed by #92 🍾.

I'm closing that issue, but feel free to re-open it if you get new errors.