Traceback on run

[iranzo]

Sometimes and not in all computers I get a traceback like this:

Traceback (most recent call last):
  File "/home/iranzo/.bin/peru", line 11, in <module>
    sys.exit(main())
  File "/home/iranzo/.local/venvs/peru/lib/python3.6/site-packages/peru/main.py", line 370, in main
    run_task(command_fn(params))
  File "/home/iranzo/.local/venvs/peru/lib/python3.6/site-packages/peru/async_helpers.py", line 29, in run_task
    return asyncio.get_event_loop().run_until_complete(coro)
  File "/usr/lib64/python3.6/asyncio/base_events.py", line 473, in run_until_complete
    return future.result()
  File "/home/iranzo/.local/venvs/peru/lib/python3.6/site-packages/peru/main.py", line 92, in do_sync
    params.runtime.sync_dir)
  File "/home/iranzo/.local/venvs/peru/lib/python3.6/site-packages/peru/imports.py", line 19, in checkout
    previous_index_file=index)
  File "/home/iranzo/.local/venvs/peru/lib/python3.6/site-packages/peru/cache.py", line 375, in export_tree
    await session.read_tree_and_stats_into_index(previous_tree)
  File "/home/iranzo/.local/venvs/peru/lib/python3.6/site-packages/peru/cache.py", line 97, in read_tree_and_stats_into_index
    await self.read_tree_into_index(tree)
  File "/home/iranzo/.local/venvs/peru/lib/python3.6/site-packages/peru/cache.py", line 94, in read_tree_into_index
    await self.git('read-tree', tree)
  File "/home/iranzo/.local/venvs/peru/lib/python3.6/site-packages/peru/cache.py", line 76, in git
    raise GitError(command, process.returncode, stdout, stderr)
peru.cache.GitError: git command "['git', '--git-dir=/home/iranzo/DEVEL/private/elegant/documentation/.peru/cache/trees', '--work-tree=/home/iranzo/DEVEL/private/elegant/documentation', 'read-tree', '737a8be24289595c1afca406acef87642a1dd3ed']" returned error code 128.
stdout: 
stderr: fatal: failed to unpack tree object 737a8be24289595c1afca406acef87642a1dd3ed

With a peru.yaml like this:

imports:
    # The dircolors file just goes at the root of our project.
    sitemap: plugins/
    assets: plugins/
    better_codeblock_line_numbering: plugins/
    better_figures_and_images: plugins/
    yuicompressor: plugins/
    elegant: themes/elegant/
    extract_toc: plugins/
    tipue_search: plugins/
    liquid_tags: plugins/
    neighbors: plugins/
    render_math: plugins/
    related_posts: plugins/
    share_post: plugins/
    series: plugins/

git module sitemap:
    url: https://github.com/getpelican/pelican-plugins.git
    pick: sitemap

git module yuicompressor:
    url: https://github.com/getpelican/pelican-plugins.git
    pick: yuicompressor

git module better_figures_and_images:
    url: https://github.com/getpelican/pelican-plugins.git
    pick: better_figures_and_images

git module better_codeblock_line_numbering:
    url: https://github.com/getpelican/pelican-plugins.git
    pick: better_codeblock_line_numbering

git module elegant:
    url: https://github.com/Pelican-Elegant/elegant.git

    # export: themes/

git module extract_toc:
    url: https://github.com/getpelican/pelican-plugins.git
    pick: extract_toc

git module tipue_search:
    url: https://github.com/getpelican/pelican-plugins.git
    pick: tipue_search

git module liquid_tags:
    url: https://github.com/getpelican/pelican-plugins.git
    pick: liquid_tags

git module neighbors:
    url: https://github.com/getpelican/pelican-plugins.git
    pick: neighbors

git module render_math:
    url: https://github.com/getpelican/pelican-plugins.git
    pick: render_math

git module related_posts:
    url: https://github.com/getpelican/pelican-plugins.git
    pick: related_posts

git module share_post:
    url: https://github.com/getpelican/pelican-plugins.git
    pick: share_post

git module series:
    url: https://github.com/getpelican/pelican-plugins.git
    pick: series

git module assets:
    url: https://github.com/getpelican/pelican-plugins.git
    pick: assets

Something wrong on my side? as the same peru.yaml works fine on other computers from time to time.

only variable set is the PERU_CACHE_DIR="~/.cache/peru", but either set or unset, I do get the same error.

Regards, Pablo

[oconnor663]

I wonder if this bug is caused by the fact that you have so many modules talking to the same git repo. Under the covers peru is going to make them share a single cached repo, and it's possible that that leads to some of the caching machinery getting confused. Do you know what steps you might've taken to hit the bug? Maybe something like "the upstream repo forcefully rebases and then I peru sync again" or something like that? So far with a simple sync I haven't been able to repro your problem.

All that said, you might be interested in the Rules section of the readme, for reducing the amount of duplication you have to write in your peru.yaml file. What you're doing here probably isn't an issue in most cases because of how caching works, but you might notice some less common cases (e.g. peru sync --no-cache) where things substantially slow down, because most of your modules end up waiting on the same cache lock and so running only one-at-a-time.

[iranzo]

Hi, thing is that that peru.yaml works fine sometimes and sometimes fail, and theoretically the upstream repo doesn't force rebases, so that's why I tried to either set or unset the peru cache dir and even clean it up to start clean again.

I've replaced the repos to rules:

imports:
    # The dircolors file just goes at the root of our project.
    elegant: themes/elegant/
    plugins: plugins/

git module elegant:
    url: https://github.com/Pelican-Elegant/elegant.git
    rev: next

git module plugins:
    url: https://github.com/getpelican/pelican-plugins.git

rule sitemap:
    pick: sitemap

rule yuicompressor:
    pick: yuicompressor

rule better_figures_and_images:
    pick: better_figures_and_images

rule better_codeblock_line_numbering:
    pick: better_codeblock_line_numbering

rule extract_toc:
    pick: extract_toc

rule tipue_search:
    pick: tipue_search

rule liquid_tags:
    pick: liquid_tags

rule neighbors:
    pick: neighbors

rule render_math:
    pick: render_math

rule related_posts:
    pick: related_posts

rule share_post:
    pick: share_post

rule series:
    pick: series

rule assets:
    pick: assets

rule post_stats:
    pick: post_stats

And cleaned the cache dir, and even tried with --no-cache

Is there any way I can try to get more debug information to help diagnosing this?

Thanks! Pablo

[iranzo]

Tried -v and got this:

 iranzo   source  ~  www  peru sync --no-cache -v === started elegant === === started plugins === === finished elegant === git fetch https://github.com/Pelican-Elegant/elegant.git

=== started elegant === === finished elegant === git fetch https://github.com/Pelican-Elegant/elegant.git

=== finished plugins === git fetch https://github.com/getpelican/pelican-plugins.git WARNING: submodule pelican-fontawesome is configured in .gitmodules, but missing in the repo

=== started plugins === === finished plugins === git fetch https://github.com/getpelican/pelican-plugins.git WARNING: submodule pelican-fontawesome is configured in .gitmodules, but missing in the repo

Traceback (most recent call last): File "/home/iranzo/.bin/peru", line 11, in sys.exit(main()) File "/home/iranzo/.local/venvs/peru/lib/python3.6/site-packages/peru/main.py", line 370, in main run_task(command_fn(params)) File "/home/iranzo/.local/venvs/peru/lib/python3.6/site-packages/peru/async_helpers.py", line 29, in run_task return asyncio.get_event_loop().run_until_complete(coro) File "/usr/lib64/python3.6/asyncio/base_events.py", line 473, in run_until_complete return future.result() File "/home/iranzo/.local/venvs/peru/lib/python3.6/site-packages/peru/main.py", line 92, in do_sync params.runtime.sync_dir) File "/home/iranzo/.local/venvs/peru/lib/python3.6/site-packages/peru/imports.py", line 19, in checkout previous_index_file=index) File "/home/iranzo/.local/venvs/peru/lib/python3.6/site-packages/peru/cache.py", line 375, in export_tree await session.read_tree_and_stats_into_index(previous_tree) File "/home/iranzo/.local/venvs/peru/lib/python3.6/site-packages/peru/cache.py", line 97, in read_tree_and_stats_into_index await self.read_tree_into_index(tree) File "/home/iranzo/.local/venvs/peru/lib/python3.6/site-packages/peru/cache.py", line 94, in read_tree_into_index await self.git('read-tree', tree) File "/home/iranzo/.local/venvs/peru/lib/python3.6/site-packages/peru/cache.py", line 76, in git raise GitError(command, process.returncode, stdout, stderr) peru.cache.GitError: git command "['git', '--git-dir=/home/iranzo/.cache/peru/trees', '--work-tree=/home/iranzo/DEVEL/private/www', 'read-tree', 'c36bf204d39d632a9a14cf8ff6b3aaf8941738c9']" returned error code 128. stdout: stderr: fatal: failed to unpack tree object c36bf204d39d632a9a14cf8ff6b3aaf8941738c9

Git clone of the repo works fine: git clone git@github.com:getpelican/pelican-plugins.git

Is maybe that submodule is listed even if not initialized and not available?

git submodule init git submodule update

does work (but that pelican-fontawesome) is not there either (but no error, etc)

[iranzo]

Seems to have worked, I removed a trailing ".peru" subdir in current folder from previous attemps and now it syncs properly even on the PERU_CACHE_DIR, closing the issue.

Thanks a lot! Pablo

[oconnor663]

Interesting. I still suspect there's a bug here, so if you ever manage to run into this again (or if you still have a .peru dir sitting around somewhere that you could tar up and send to me), let me know.