WindowsError: [Error 32] The process cannot access the file because it is being used by another process

[PrzemekWirkus]

Description

Found issue with yotta 0.14.1 while using it with in-house CI system:

WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:\\Windows\\system32\\config\\systemprofile/.yotta\\cache\\1bea8e6b3fef24a15207945cd81f9be8bd88872422bf56ece08ee96d3dcea6f6'

Jobs affected (Examples):

• http://e108747.cambridge.arm.com:8080/job/ybt/250/console
• http://e108747.cambridge.arm.com:8080/job/ybt/253/console

  Traceback

$ yotta build
info: get versions for core-util
info: download core-util@1.5.2 from the public module registry
info: get versions for greentea-client
info: download greentea-client@0.1.4 from the public module registry
info: get versions for unity
info: download unity@2.1.0 from the public module registry
info: get versions for ualloc
info: download ualloc@1.0.3 from the public module registry
info: get versions for cmsis-core
info: download cmsis-core@1.1.2 from the public module registry
info: get versions for dlmalloc
info: download dlmalloc@1.0.0 from the public module registry
Fatal Exception, yotta=0.14.1
Traceback (most recent call last):
  File "c:\python27\Lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\python27\Lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\jslv3_1\ws\ybt@2\venv\Scripts\yotta.exe\__main__.py", line 9, in <module>
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\main.py", line 201, in main
    status = args.command(args, following_args)
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\build.py", line 46, in execCommand
    status = installAndBuild(args, following_args)
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\build.py", line 98, in installAndBuild
    install_status = install.execCommand(args, [])
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\install.py", line 58, in execCommand
    return installDeps(args, c)
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\install.py", line 126, in installDeps
    test = {'own':'toplevel', 'all':True, 'none':False}[args.install_test_deps]
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\component.py", line 655, in satisfyDependenciesRecursive
    test = test
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\component.py", line 440, in __getDependenciesRecursiveWithProvider
    _processed = _processed
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\component.py", line 440, in __getDependenciesRecursiveWithProvider
    _processed = _processed
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\component.py", line 419, in __getDependenciesRecursiveWithProvider
    test = test
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\component.py", line 319, in __getDependenciesWithProvider
    satisfyDep, specs
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\component.py", line 295, in satisfyDep
    self
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\component.py", line 640, in provider
    inherit_shrinkwrap = dep_of.getShrinkwrap()
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\access.py", line 317, in satisfyVersionByInstalling
    name, version_required, install_into, v, type=type, inherit_shrinkwrap = inherit_shrinkwrap
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\access.py", line 328, in _satisfyVersionByInstallingVersion
    version.unpackInto(working_directory)
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\registry_access.py", line 447, in unpackInto
    _getTarball(self.url, directory, self.sha256)
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\registry_access.py", line 100, in wrapped
    return fn(*args, **kwargs)
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\registry_access.py", line 185, in wrapped
    return fn(*args, **kwargs)
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\registry_access.py", line 168, in wrapped
    return fn(*args, **kwargs)
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\registry_access.py", line 149, in wrapped
    return fn(*args, **kwargs)
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\registry_access.py", line 296, in _getTarball
    origin_info = {'url':url}
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\access_common.py", line 137, in wrapped
    pruneCache()
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\access_common.py", line 127, in pruneCache
    removeFromCache(f)
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\access_common.py", line 190, in removeFromCache
    fsutils.rmF(f)
  File "c:\jslv3_1\ws\ybt@2\venv\lib\site-packages\yotta\lib\fsutils.py", line 24, in rmF
    os.remove(path)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:\\Windows\\system32\\config\\systemprofile/.yotta\\cache\\1bea8e6b3fef24a15207945cd81f9be8bd88872422bf56ece08ee96d3dcea6f6'

[PrzemekWirkus]

@autopulated James, this issue is giving us pretty nasty CI behavior. If you have time please have a look at it. Thanks!

[0xc0170]

I experience this from time to time. My usual workaround is to execute the command multiple times. I have similar WindowsErrors when I do yt clean or yt up. Most of the time, 2x/3x does the trick

[PrzemekWirkus]

@0xc0170 It';s critical that build system doesn't fail like that. Especially if we run yotta in continuous Integration +100 times per nightly.

[PrzemekWirkus]

Mirrored to IOTTOO-2166.

[autopulated]

I believe this is caused by antivirus software locking files for a short period after they're created.

yotta's recursive-remove function already has a workaround for this (retry the deletion in a loop...), so should be fairly straightforward to copy that over to the non-recursive remove.

As a short-term amelioration you could try increasing the cache size that you have set up (maxCachedModules:<number> in ~/.yotta/config.json), which should cause fewer items to be removed from the cache.

[PrzemekWirkus]

@autopulated James, dozens of instances of yotta in our CI try to access /.yotta/cache/* at the same time. This might not be antivirus thing IMHO, this might be a concurrent access to files. Do we have in yotta (and in yotta.lib.access_common ) any locking mechanism on top of /.yotta/cache so when multiple instances try to access it they do not compete ?

[PrzemekWirkus]

I also would like yotta to work with maxCachedModules set to any number correctly.

[autopulated]

Multiple instances should share the same cache, and it should be safe.

There is no locking mechanism, per se, but the cache is intended to be safe for multiple instance to use (files are only added to the cache via an atomic 'rename' operation once they have been downloaded elsewhere), and removed by an atomic 'unlink'.

[PrzemekWirkus]

@autopulated James, those are issues from today's nightly:

WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:\\Windows\\system32\\config\\systemprofile/.yotta\\cache\\tmp3gjhrd'

WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:\\Windows\\system32\\config\\systemprofile/.yotta\\cache\\f4c42da210c12354d4f17fa7f1d80953db0b08595ceb3ade466ac1da4d0c2a19'

WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:\\Windows\\system32\\config\\systemprofile/.yotta\\cache\\e980c9c9c306b96e3f98402a038d58f27e0c87a6adeea71c73f9f098e0601154'

WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:\\Windows\\system32\\config\\systemprofile/.yotta\\cache\\1982ab1f0124dc82bff025a1d757af81b8d8398e258b5259f28f1f824d659c4e'

WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:\\Windows\\system32\\config\\systemprofile/.yotta\\cache\\5612353b4bda7f716823f0e717805ed72ebdd830739f3848ed55794ac642c506'

WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:\\Windows\\system32\\config\\systemprofile/.yotta\\cache\\tmpdkquaw'

WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:\\Windows\\system32\\config\\systemprofile/.yotta\\cache\\d1fb276cb12ebfc9b62e4d8ed647f82a2971a04ed7ae35fc2a6da017a937ad03'

WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:\\Windows\\system32\\config\\systemprofile/.yotta\\cache\\1736937df92125787af1f1be88c9570c0419d568ee545450abb6106c48b076cd'

WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:\\Windows\\system32\\config\\systemprofile/.yotta\\cache\\b01a427decd3db247ebdf3f46cebbebd63b35c4577489f7da2b7a9fd047b78ed'

[PrzemekWirkus]

All have the same traceback:

Traceback (most recent call last):
  File "c:\python27\Lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\python27\Lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\jslv3_1\ws\yb@9\venv\Scripts\yotta.exe\__main__.py", line 9, in <module>
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\main.py", line 201, in main
    status = args.command(args, following_args)
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\build.py", line 46, in execCommand
    status = installAndBuild(args, following_args)
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\build.py", line 98, in installAndBuild
    install_status = install.execCommand(args, [])
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\install.py", line 58, in execCommand
    return installDeps(args, c)
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\install.py", line 126, in installDeps
    test = {'own':'toplevel', 'all':True, 'none':False}[args.install_test_deps]
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\component.py", line 655, in satisfyDependenciesRecursive
    test = test
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\component.py", line 440, in __getDependenciesRecursiveWithProvider
    _processed = _processed
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\component.py", line 440, in __getDependenciesRecursiveWithProvider
    _processed = _processed
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\component.py", line 419, in __getDependenciesRecursiveWithProvider
    test = test
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\component.py", line 319, in __getDependenciesWithProvider
    satisfyDep, specs
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\component.py", line 295, in satisfyDep
    self
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\component.py", line 640, in provider
    inherit_shrinkwrap = dep_of.getShrinkwrap()
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\access.py", line 317, in satisfyVersionByInstalling
    name, version_required, install_into, v, type=type, inherit_shrinkwrap = inherit_shrinkwrap
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\access.py", line 328, in _satisfyVersionByInstallingVersion
    version.unpackInto(working_directory)
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\registry_access.py", line 447, in unpackInto
    _getTarball(self.url, directory, self.sha256)
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\registry_access.py", line 100, in wrapped
    return fn(*args, **kwargs)
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\registry_access.py", line 185, in wrapped
    return fn(*args, **kwargs)
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\registry_access.py", line 168, in wrapped
    return fn(*args, **kwargs)
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\registry_access.py", line 149, in wrapped
    return fn(*args, **kwargs)
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\registry_access.py", line 296, in _getTarball
    origin_info = {'url':url}
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\access_common.py", line 137, in wrapped
    pruneCache()
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\access_common.py", line 127, in pruneCache
    removeFromCache(f)
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\access_common.py", line 190, in removeFromCache
    fsutils.rmF(f)
  File "c:\jslv3_1\ws\yb@9\venv\lib\site-packages\yotta\lib\fsutils.py", line 24, in rmF
    os.remove(path)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:\\Windows\\system32\\config\\systemprofile/.yotta\\cache\\f4c42da210c12354d4f17fa7f1d80953db0b08595ceb3ade466ac1da4d0c2a19'

Are you able to provide any type of fix for this this os.remove(path) behavior ?

[autopulated]

what is maxCachedModules set to on your build server?

[mazimkhan]

Can setting YOTTA_PREFIX make these cache files to be created in separate directory. If true, we can have a directory in Jenkins workspace and isolate concurrent builds.

[autopulated]

Setting YOTTA_USER_SETTINGS_DIR will change the cache directory

[mazimkhan]

Will changing user settings directory require yotta login? If yes then can't use in automation.

[autopulated]

yotta 0.14.2 is now live, which includes the fix for this. Thanks for your patience!