awscli2: broken by recent upgrade of python3Packages.awscrt

[neilmayhew]

Describe the bug awscli2 fails to build.

To Reproduce Steps to reproduce the behavior:

$ git describe
21.11-pre-7584-g520da1d00f2
$ nix-build -A awscli2 .
...
ERROR: Could not find a version that satisfies the requirement awscrt==0.11.13 (from awscli)
ERROR: No matching distribution found for awscrt==0.11.13

Expected behavior awscli2 should build.

Additional context

This fixes it:

diff --git a/pkgs/tools/admin/awscli2/default.nix b/pkgs/tools/admin/awscli2/default.nix
index 0824a26855d..b52cc80b7b1 100644
--- a/pkgs/tools/admin/awscli2/default.nix
+++ b/pkgs/tools/admin/awscli2/default.nix
@@ -41,6 +41,7 @@ with py.pkgs; buildPythonApplication rec {
   };

   postPatch = ''
+    substituteInPlace setup.py --replace "awscrt==0.11.13" "awscrt>=0.11.13"
     substituteInPlace setup.py --replace "colorama>=0.2.5,<0.4.4" "colorama>=0.2.5"
     substituteInPlace setup.py --replace "cryptography>=3.3.2,<3.4.0" "cryptography>=3.3.2"
     substituteInPlace setup.py --replace "docutils>=0.10,<0.16" "docutils>=0.10"

Notify maintainers

@bhipple @davegallant

Metadata

• system: "x86_64-linux"
• host os: Linux 5.4.122, NixOS, 20.09.4377.1b42ce6bead (Nightingale)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.4pre20210503_6d2553a
• channels(root): "nixos-20.09.4377.1b42ce6bead"
• channels(user): "nixpkgs-21.11pre299363.e717ab43895"
• nixpkgs: $HOME/.nix-defexpr/channels/nixpkgs

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[pecigonzalo]

I fixed this temporarly doing:

self: super:
{
  awscli2 = super.awscli2.overrideAttrs (old: {
    postPatch = ''
      substituteInPlace setup.py --replace "awscrt==0.11.13" "awscrt>=0.11.13"
      substituteInPlace setup.py --replace "colorama>=0.2.5,<0.4.4" "colorama>=0.2.5"
      substituteInPlace setup.py --replace "cryptography>=3.3.2,<3.4.0" "cryptography>=3.3.2"
      substituteInPlace setup.py --replace "docutils>=0.10,<0.16" "docutils>=0.10"
      substituteInPlace setup.py --replace "ruamel.yaml>=0.15.0,<0.16.0" "ruamel.yaml>=0.15.0"
      substituteInPlace setup.py --replace "wcwidth<0.2.0" "wcwidth"
    '';
  });
}

in my ~/.config/nixpkgs/overlays/awscli2.nix.

(I use home-manager)

[davegallant]

This should be fixed now with https://github.com/NixOS/nixpkgs/pull/128273.

All of the version specifiers are now removed during the substituteInPlace, so it should just grab whatever python package nixpkgs has. Hopefully this reduces the amount of build failures going forward.

[favadi]

I started seeing build failure for awscli on macOS after updating nixpkgs-unstable today.

======================================================================
FAIL: test_upload_move_using_crt_client (tests.functional.s3.test_mv_command.TestMvWithCRTClient)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/private/tmp/nix-build-awscli2-2.2.14.drv-0/source/tests/functional/s3/test_mv_command.py", line 252, in test_upload_move_using_crt_client
    self.run_command(cmdline)
  File "/private/tmp/nix-build-awscli2-2.2.14.drv-0/source/tests/functional/s3/__init__.py", line 375, in run_command
    self.assertEqual(
AssertionError: 255 != 0 : Expected rc of 0 instead got 255 with stderr message:
30 (AWS_ERROR_PRIORITY_QUEUE_EMPTY): Attempt to pop an item from an empty queue.

-------------------- >> begin captured logging << --------------------
awscli.plugin: DEBUG: Importing plugin __builtin__: awscli.handlers
awscli.plugin: DEBUG: Initializing plugin __builtin__: <module 'awscli.handlers' from '/private/tmp/nix-build-awscli2-2.2.14.drv-0/source/awscli/handlers.py'>
awscli.plugin: DEBUG: cli_legacy_plugin_path not defined in plugin section. Not importing additional plugins.
botocore.hooks: DEBUG: Event building-command-table.main: calling handler <function add_s3 at 0x10de73c10>
botocore.hooks: DEBUG: Event building-command-table.main: calling handler <function add_ddb at 0x10e7b4790>
botocore.hooks: DEBUG: Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
botocore.hooks: DEBUG: Event building-command-table.main: calling handler <function change_name at 0x10e7b40d0>
botocore.hooks: DEBUG: Event building-command-table.main: calling handler <function change_name at 0x10e7b45e0>
botocore.hooks: DEBUG: Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x10e7f6820>
botocore.hooks: DEBUG: Event building-command-table.main: calling handler <function add_history_commands at 0x10da6bf70>
botocore.hooks: DEBUG: Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
botocore.hooks: DEBUG: Event building-command-table.main: calling handler <function add_waiters at 0x10ded7040>
botocore.loaders: DEBUG: Loading JSON file: /private/tmp/nix-build-awscli2-2.2.14.drv-0/source/awscli/data/cli.json
botocore.hooks: DEBUG: Event top-level-args-parsed: calling handler <function resolve_types at 0x10e137a60>
botocore.hooks: DEBUG: Event top-level-args-parsed: calling handler <function no_sign_request at 0x10e137ca0>
botocore.hooks: DEBUG: Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x10e137c10>
botocore.hooks: DEBUG: Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x10e137dc0>
botocore.hooks: DEBUG: Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x10e137d30>
botocore.hooks: DEBUG: Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x18ab87140>
botocore.hooks: DEBUG: Event session-initialized: calling handler <function add_timestamp_parser at 0x10dea9700>
botocore.hooks: DEBUG: Event session-initialized: calling handler <function register_uri_param_handler at 0x10dfe7dc0>
botocore.hooks: DEBUG: Event session-initialized: calling handler <function add_binary_formatter at 0x10dfe2dc0>
botocore.hooks: DEBUG: Event session-initialized: calling handler <function no_pager_handler at 0x10d415790>
botocore.hooks: DEBUG: Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x10d61d700>
botocore.utils: DEBUG: IMDS ENDPOINT: http://169.254.169.254/
botocore.hooks: DEBUG: Event session-initialized: calling handler <function attach_history_handler at 0x10da6be50>
botocore.hooks: DEBUG: Event session-initialized: calling handler <function inject_json_file_cache at 0x10d6271f0>
botocore.hooks: DEBUG: Event building-command-table.s3: calling handler <function add_waiters at 0x10ded7040>
botocore.hooks: DEBUG: Event load-cli-arg.custom.s3.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event building-command-table.s3_mv: calling handler <function add_waiters at 0x10ded7040>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.paths: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.dryrun: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event process-cli-arg.custom.mv: calling handler <awscli.argprocess.ParamShorthandParser object at 0x18de2ccd0>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.quiet: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event process-cli-arg.custom.mv: calling handler <awscli.argprocess.ParamShorthandParser object at 0x18de2ccd0>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.acl: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.follow-symlinks: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event process-cli-arg.custom.mv: calling handler <awscli.argprocess.ParamShorthandParser object at 0x18de2ccd0>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.sse: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.sse-c: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.sse-c-key: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.sse-kms-key-id: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.sse-c-copy-source: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.sse-c-copy-source-key: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.storage-class: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.grants: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.website-redirect: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.content-type: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.cache-control: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.content-disposition: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.content-encoding: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.content-language: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.expires: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.source-region: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.only-show-errors: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event process-cli-arg.custom.mv: calling handler <awscli.argprocess.ParamShorthandParser object at 0x18de2ccd0>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.ignore-glacier-warnings: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event process-cli-arg.custom.mv: calling handler <awscli.argprocess.ParamShorthandParser object at 0x18de2ccd0>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.force-glacier-transfer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event process-cli-arg.custom.mv: calling handler <awscli.argprocess.ParamShorthandParser object at 0x18de2ccd0>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.request-payer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.metadata: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.copy-props: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event process-cli-arg.custom.mv: calling handler <awscli.argprocess.ParamShorthandParser object at 0x18de2ccd0>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.metadata-directive: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.hooks: DEBUG: Event load-cli-arg.custom.mv.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x18da51490>
botocore.credentials: DEBUG: Looking for credentials via: env
botocore.credentials: INFO: Found credentials in environment variables.
botocore.loaders: DEBUG: Loading JSON file: /nix/store/vgxp453j0y9aylyyvy0bp745jm642a2d-python3.8-botocore-2.0.0dev122/lib/python3.8/site-packages/botocore/data/endpoints.json
botocore.hooks: DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x10cb87280>
botocore.loaders: DEBUG: Loading JSON file: /nix/store/vgxp453j0y9aylyyvy0bp745jm642a2d-python3.8-botocore-2.0.0dev122/lib/python3.8/site-packages/botocore/data/s3/2006-03-01/service-2.json
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x10c79a700>
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x10c79a4c0>
botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60)
botocore.hooks: DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x10cb87280>
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x10c79a700>
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x10c79a4c0>
botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60)
botocore.hooks: DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x10cb87280>
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x10c79a700>
botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x10c79a4c0>
botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60)
awscli.clidriver: DEBUG: Exception caught in main()
Traceback (most recent call last):
  File "/private/tmp/nix-build-awscli2-2.2.14.drv-0/source/awscli/clidriver.py", line 459, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/private/tmp/nix-build-awscli2-2.2.14.drv-0/source/awscli/customizations/commands.py", line 197, in __call__
    return self.subcommand_table[parsed_args.subcommand](remaining,
  File "/private/tmp/nix-build-awscli2-2.2.14.drv-0/source/awscli/customizations/commands.py", line 191, in __call__
    rc = self._run_main(parsed_args, parsed_globals)
  File "/private/tmp/nix-build-awscli2-2.2.14.drv-0/source/awscli/customizations/s3/subcommands.py", line 716, in _run_main
    transfer_manager = self._get_transfer_manager(
  File "/private/tmp/nix-build-awscli2-2.2.14.drv-0/source/awscli/customizations/s3/subcommands.py", line 758, in _get_transfer_manager
    return TransferManagerFactory(self._session).create_transfer_manager(
  File "/private/tmp/nix-build-awscli2-2.2.14.drv-0/source/awscli/customizations/s3/factory.py", line 65, in create_transfer_manager
    return self._create_crt_transfer_manager(params, runtime_config)
  File "/private/tmp/nix-build-awscli2-2.2.14.drv-0/source/awscli/customizations/s3/factory.py", line 80, in _create_crt_transfer_manager
    self._create_crt_client(params, runtime_config),
  File "/private/tmp/nix-build-awscli2-2.2.14.drv-0/source/awscli/customizations/s3/factory.py", line 103, in _create_crt_client
    return create_s3_crt_client(**create_crt_client_kwargs)
  File "/nix/store/www8il7nnl43242y0qdj7a632109n5q6-python3.8-s3transfer-0.4.2/lib/python3.8/site-packages/s3transfer/crt.py", line 105, in create_s3_crt_client
    event_loop_group = EventLoopGroup(num_threads)
  File "/nix/store/lfimhaxfnmj4h0ck41pd6i6ax5c8hwpc-python3.8-awscrt-0.11.24/lib/python3.8/site-packages/awscrt/io.py", line 84, in __init__
    self._binding = _awscrt.event_loop_group_new(num_threads, is_pinned, cpu_group, on_shutdown)
RuntimeError: 30 (AWS_ERROR_PRIORITY_QUEUE_EMPTY): Attempt to pop an item from an empty queue.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_delete_using_crt_client (tests.functional.s3.test_rm_command.TestRmWithCRTClient)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/private/tmp/nix-build-awscli2-2.2.14.drv-0/source/tests/functional/s3/test_rm_command.py", line 80, in test_delete_using_crt_client
    self.run_command(cmdline)

Not sure if related to this issue.