search

ansible-community / antsibull-changelog

Changelog generator for ansible, ansible-base, and collections.
https://ansible.readthedocs.io/projects/antsibull-changelog/
GNU General Public License v3.0
36 stars 28 forks source link

refreshing plugin cache seems to get stuck forever #172

Closed ssbarnea closed 2 weeks ago

ssbarnea commented 3 weeks ago

On MacOs when I did run release command it seems to remain stuck at something forever. After adding -v I found out that this must be related to plugin cache.

As this operation seems to take forever (minutes!), we need to ensure we display some kind of progress information.

$ antsibull-changelog release -vv --version 1.4.7`
INFO refreshing plugin cache`
^CTraceback (most recent call last):
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/bin/antsibull-changelog", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/site-packages/antsibull_changelog/cli.py", line 879, in main
    return run(sys.argv)
           ^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/site-packages/antsibull_changelog/cli.py", line 334, in run
    return arguments.func(arguments)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/site-packages/antsibull_changelog/cli.py", line 659, in command_release
    plugins = load_plugins(
              ^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/site-packages/antsibull_changelog/plugins.py", line 590, in load_plugins
    plugins_data = _refresh_plugin_cache(
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/site-packages/antsibull_changelog/plugins.py", line 535, in _refresh_plugin_cache
    _load_collection_plugins_2_13(plugins_data, paths, collection_details)
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/site-packages/antsibull_changelog/plugins.py", line 450, in _load_collection_plugins_2_13
    with CollectionCopier(
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/site-packages/antsibull_changelog/plugins.py", line 396, in __enter__
    shutil.copytree(self.paths.base_dir, collection_dir, symlinks=True)
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 600, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 536, in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 600, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 536, in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 600, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 536, in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 600, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 536, in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 600, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 536, in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 600, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 536, in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 600, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 536, in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 600, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 536, in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 600, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 536, in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 600, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 536, in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 600, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 536, in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 600, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 536, in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 600, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 540, in _copytree
    copy_function(srcobj, dstname)
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 475, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 266, in copyfile
    _fastcopy_fcopyfile(fsrc, fdst, posix._COPYFILE_DATA)
  File "/Users/ssbarnea/.asdf/installs/python/3.12.3/lib/python3.12/shutil.py", line 103, in _fastcopy_fcopyfile
    posix._fcopyfile(infd, outfd, flags)
KeyboardInterrupt

Any ideas?

felixfontein commented 3 weeks ago

What antsibull-changelog does is copying the collection's directory to a temp directory structure (ending with ansible_collections/<namespace>/<name>) using shutil.copytree(collection_source_directory, temp_directory, symlinks=True). My guess is that you have a lot of things lying around in your collection's checkout so that the shutil.copytree takes a very long time.

felixfontein commented 2 weeks ago

175 allows to not copy the files ignored by git.