AssertionError: cannot start a process twice

[echoboomer]

First time running this locally - it's a great tool, thanks for writing it! I'm using the latest version.

I encountered this issue on my first try:

$ python3 py-Classic-Resource-Finder.py --profile production
Checking the Classic platform status in us-east-1
Checking for EIPs in us-east-1
Checking for Classic EC2 Instances in us-east-1
Checking for Classic Security Groups in us-east-1
Checking for VPCs with ClassicLink enabled in us-east-1
Checking for AutoScaling Groups configured for Classic in us-east-1
Checking for Classic Load Balancers running in EC2-Classic in us-east-1
Checking for Classic RDS Instances in us-east-1
Checking for Classic ElastiCache clusters in us-east-1
Checking for Classic Redshift clusters in us-east-1
Checking for Classic Elastic BeanStalk Environments in us-east-1
Checking for Classic EMR clusters in us-east-1
Checking for Classic OpsWorks stacks in us-east-1
Checking for Classic Data Pipelines in us-east-1
Traceback (most recent call last):
  File "/Users/scott/src/external/ec2-classic-resource-finder/py-Classic-Resource-Finder.py", line 847, in <module>
    main(argparser(sys.argv[1:]))
  File "/Users/scott/src/external/ec2-classic-resource-finder/py-Classic-Resource-Finder.py", line 841, in main
    loopregions(classicregions, datapipelineregions, creddict)
  File "/Users/scott/src/external/ec2-classic-resource-finder/py-Classic-Resource-Finder.py", line 668, in loopregions
    process.start()
  File "/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 115, in start
    assert self._popen is None, 'cannot start a process twice'
AssertionError: cannot start a process twice

As you can see above, I'm running Python 3.9.9:

$ python3 --version
Python 3.9.9

I did get partial results from the single region and the errors file was empty after switching to my mfa profile.

Any hints?

[TheScottMo]

Is this only happening when you're using a profile or does it occur with other authentication methods? also, do you only and always get this error under Data Pipelines, or does it happen under other services at all?

[echoboomer]

Profile is the only method I've tried so far, I haven't tried with role assumption. It always happens right at the end under data pipelines, so each time I run the script the output is just like that.

[TheScottMo]

Ok, thank you. Let me do some testing in my environment and see if I can replicate this.

[sqlbot]

fwiw, I'm experiencing essentially the same reported behavior (with python 3.3.8 on Ubuntu 20.04.3 on WSL2 on Windows 10 21H1 build 19043.1348) when I use --profile...

$ python3 py-Classic-Resource-Finder.py --profile myprofile
Checking the Classic platform status in us-east-1
Checking for EIPs in us-east-1
Checking for Classic EC2 Instances in us-east-1
-- snip --
Checking for Classic Data Pipelines in us-east-1
Traceback (most recent call last):
  File "../py-Classic-Resource-Finder.py", line 847, in <module>
    main(argparser(sys.argv[1:]))
  File "../py-Classic-Resource-Finder.py", line 841, in main
    loopregions(classicregions, datapipelineregions, creddict)
  File "../py-Classic-Resource-Finder.py", line 668, in loopregions
    process.start()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 115, in start
    assert self._popen is None, 'cannot start a process twice'
AssertionError: cannot start a process twice

...however, the code seems to work correctly when the same profile is active via environment variable.

$ env AWS_PROFILE=myprofile python3 py-Classic-Resource-Finder.py
-- snip --
Checking for Classic OpsWorks stacks in ap-southeast-2
Checking for Classic OpsWorks stacks in ap-southeast-1
Checking for Classic Data Pipelines in ap-southeast-2
finished

Out of curiosity, I tried specifying the same profile in both places.

$ env AWS_PROFILE=myprofile python3 py-Classic-Resource-Finder.py --profile myprofile

...but this also fails, with the same error that occurs when using --profile alone.

In my environment, I don't use a [default] section in ~/.aws/config (to avoid inadvertently/implicitly using a set of credentials or region) but adding that section with a valid key and secret also did not seem to make any difference -- the code still fails with the same error when using --profile.

[TheScottMo]

Hello, thank you for adding this information. I think I have found the cause of this error. I am working towards a fix, in the interim if you set your environmental variables to use the active profile and dont use the --profile flag, that should work until I can get this fix pushed.

[echoboomer]

Can confirm using the environment variable to set profile works. Thanks!

[nngabriel]

Can confirm using the environment variable to set profile works. Thanks!

I was having the same issue. But setting the environment variable to the right profile did the trick. Of course without using the -p option. Thanks Scott

[TheScottMo]

Turns out a few extra tabs in the wrong places make a whole world of difference. I have pushed a fix which should resolve this issue. (NOTE: this bug also affected organizational runs as well, although they saw a much less verbose error)