Invalid bucket name AWS

[richooms]

Hi there,

When trying to run some tests to see if the benchmark is running on AWS. When trying to do so I get the error: Invalid bucket name "thesisbenchooms\ec2": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$"

The \ec2 part is added automatically to the name I edited in the config file as described in the instructions. For more information the complete error log is copied below.

fatal error: Parameter validation failed: [ 113.140682] cloud-init[1435]: Invalid bucket name "thesisbenchooms\ec2": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" [ 113.629291] cloud-init[1435]: fatal error: Parameter validation failed: [ 113.631229] cloud-init[1435]: Invalid bucket name "thesisbenchooms\ec2": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" [ 114.624355] cloud-init[1435]: RunningRandomForeston/s3bucket/user/test.yamlbenchmarks inlocal` mode. [ 114.662194] cloud-init[1435]: Loading frameworks definitions from /repo/resources/frameworks.yaml.

[ 114.678772] cloud-init[1435]: Incorrect benchmark name or path /s3bucket/user/test.yaml, name not available in /repo/resources/benchmarks. [ 115.725435] cloud-init[1435]: Running RandomForest on /s3bucket/user/test.yaml benchmarks in local mode. [ 115.762451] cloud-init[1435]: Loading frameworks definitions from /repo/resources/frameworks.yaml.

[ 115.778351] cloud-init[1435]: Incorrect benchmark name or path /s3bucket/user/test.yaml, name not available in /repo/resources/benchmarks. [ 116.352902] cloud-init[1435]: upload failed: ../s3bucket/output/logs/runbenchmark_20190618T075011.log to s3://thesisbenchooms\ec2/aws_randomforest_test_20190618T074743\aws_test_allrandomforest\output/logs/runbenchmark20190618T075011.log Parameter validation failed: [ 116.354415] cloud-init[1435]: Invalid bucket name "thesisbenchooms\ec2": Bucket name must match the regex "^[a-zA-Z0-9.-]{1,255}$" [ 116.362361] cloud-init[1435]: Completed 8.9 KiB/~9.3 KiB (0 Bytes/s) with ~1 file(s) remaining (calculating...) upload failed: ../s3bucket/output/logs/runbenchmark_20190618T075011_full.log to s3://thesisbenchooms\ec2/aws_randomforest_test_20190618T074743\aws_test_all__randomforest\output/logs/runbenchmark_20190618T075011full.log Parameter validation failed: [ 116.363474] cloud-init[1435]: Invalid bucket name "thesisbenchooms\ec2": Bucket name must match the regex "^[a-zA-Z0-9.-]{1,255}$" [ 116.373777] cloud-init[1435]: Completed 9.3 KiB/~18.4 KiB (0 Bytes/s) with ~1 file(s) remaining (calculating...) upload failed: ../s3bucket/output/logs/runbenchmark_20190618T075012.log to s3://thesisbenchooms\ec2/aws_randomforest_test_20190618T074743\aws_test_allrandomforest\output/logs/runbenchmark20190618T075012.log Parameter validation failed: [ 116.373937] cloud-init[1435]: Invalid bucket name "thesisbenchooms\ec2": Bucket name must match the regex "^[a-zA-Z0-9.-]{1,255}$" [ 116.386116] cloud-init[1435]: Completed 18.4 KiB/~18.4 KiB (0 Bytes/s) with ~0 file(s) remaining (calculating...) upload failed: ../s3bucket/output/logs/runbenchmark_20190618T075012_full.log to s3://thesisbenchooms\ec2/aws_randomforest_test_20190618T074743\aws_test_all__randomforest\output/logs/runbenchmark_20190618T075012full.log Parameter validation failed: [ 116.386213] cloud-init[1435]: Invalid bucket name "thesisbenchooms\ec2": Bucket name must match the regex "^[a-zA-Z0-9.-]{1,255}$" [ 116.386264] cloud-init[1435]: Completed 18.7 KiB/~18.7 KiB (0 Bytes/s) with ~0 file(s) remaining (calculating...)`

[sebhrusen]

Hi @Stefanenrichard, sorry, this looks like another incompatibility issue on Windows. I just posted a fix proposal above, can you please try with that branch?

Also, FYI, any property defined in resources/config.yaml can be overridden locally: you just need to create an empty config.yaml file in your ~/.config/automlbenchmark directory. For example, to get rid of the ec2/ key on the s3 bucket, your config.yaml file could look as follow:

---

aws:
  s3:
    bucket: thesisbenchooms
    root_key: ""

Although, given the fix branch above, I bet this would not completely solve your issue, that's why I'd prefer you to try the branch: I'll merge it if you confirm it works, thanks.

[richooms]

Hi @seb-h2o,

Thanks again for the fast reply. The /ec2 part is gone now, but there still is a similar error. When running python runbenchmark.py constantpredictor test -m aws. I still get an error on the bucket name:

short fatal error: Parameter validation failed: [ 110.625675] cloud-init[1401]: Invalid bucket name "thesisbenchooms\aws_constantpredictor_test_20190618T141330\input": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$"

long fatal error: Parameter validation failed: [ 111.130638] cloud-init[1401]: Invalid bucket name "thesisbenchooms\aws_constantpredictor_test_20190618T141330\user": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" [ 112.157124] cloud-init[1401]: Runningconstantpredictoron/s3bucket/user/test.yamlbenchmarks inlocal` mode. [ 112.194976] cloud-init[1401]: Loading frameworks definitions from /repo/resources/frameworks.yaml.

[ 112.212106] cloud-init[1401]: Incorrect benchmark name or path /s3bucket/user/test.yaml, name not available in /repo/resources/benchmarks. [ 113.306598] cloud-init[1401]: Running constantpredictor on /s3bucket/user/test.yaml benchmarks in local mode. [ 113.345137] cloud-init[1401]: Loading frameworks definitions from /repo/resources/frameworks.yaml.

[ 113.362359] cloud-init[1401]: Incorrect benchmark name or path /s3bucket/user/test.yaml, name not available in /repo/resources/benchmarks. [ 113.961157] cloud-init[1401]: upload failed: ../s3bucket/output/logs/runbenchmark_20190618T141545.log to s3://thesisbenchooms\aws_constantpredictor_test_20190618T141330\aws_test_allconstantpredictor\output/logs/runbenchmark_20190618T141545.log Parameter validation failed: [ 113.967432] cloud-init[1401]: Invalid bucket name "thesisbenchooms\aws_constantpredictor_test_20190618T141330\aws_test_allconstantpredictor\output": Bucket name must match the regex "^[a-zA-Z0-9.-_]{1,255}$" [ 113.972907] cloud-init[1401]: Completed 8.9 KiB/~9.3 KiB (0 Bytes/s) with ~1 file(s) remaining (calculating...) upload failed: ../s3bucket/output/logs/runbenchmark_20190618T141545_full.log to s3://thesisbenchooms\aws_constantpredictor_test_20190618T141330\aws_test_allconstantpredictor\output/logs/runbenchmark_20190618T141545_full.log Parameter validation failed: [ 113.981180] cloud-init[1401]: Invalid bucket name "thesisbenchooms\aws_constantpredictor_test_20190618T141330\aws_test_all_constantpredictor\output": Bucket name must match the regex "^[a-zA-Z0-9.-]{1,255}$" [ 113.981322] cloud-init[1401]: Completed 9.3 KiB/~18.4 KiB (0 Bytes/s) with ~1 file(s) remaining (calculating...) upload failed: ../s3bucket/output/logs/runbenchmark_20190618T141546.log to s3://thesisbenchooms\aws_constantpredictor_test_20190618T141330\aws_test_allconstantpredictor\output/logs/runbenchmark_20190618T141546.log Parameter validation failed: [ 113.981408] cloud-init[1401]: Invalid bucket name "thesisbenchooms\aws_constantpredictor_test_20190618T141330\aws_test_allconstantpredictor\output": Bucket name must match the regex "^[a-zA-Z0-9.-_]{1,255}$" [ 113.981563] cloud-init[1401]: Completed 18.4 KiB/~18.7 KiB (0 Bytes/s) with ~1 file(s) remaining (calculating...) upload failed: ../s3bucket/output/logs/runbenchmark_20190618T141546_full.log to s3://thesisbenchooms\aws_constantpredictor_test_20190618T141330\aws_test_allconstantpredictor\output/logs/runbenchmark_20190618T141546_full.log Parameter validation failed: [ 113.981657] cloud-init[1401]: Invalid bucket name "thesisbenchooms\aws_constantpredictor_test_20190618T141330\aws_test_all_constantpredictor\output": Bucket name must match the regex "^[a-zA-Z0-9.-]{1,255}$"`

[sebhrusen]

@Stefanenrichard did you try to checkout the fix branch fix/22/aws_wrong_url_on_win ? It looks like you've been overriding the config.yaml as I suggested to remove the ec2\ part, but I think only the fix branch can fully remove the backslashes in the url.

[richooms]

@seb-h2o I tried both. I have pulled that branch and replaced the config.yaml in the ~\.config\automlbenchmark from the examples\aws as suggested in the . Although that file seems pretty empty.

I now get (venv) C:\Users\riooms\OneDrive-DeloitteO365D)\Thesis\Benchmark\fixse\automlbenchmark>python runbenchmark.py tpot -m aws Runningtpotontestbenchmarks inawsmode. Loading frameworks definitions from C:\Users\riooms\OneDrive-DeloitteO365D)\Thesis\Benchmark\fixse\automlbenchmark\resources\frameworks.yaml. Loading benchmark definitions from C:\Users\riooms\OneDrive-DeloitteO365D)\Thesis\Benchmark\fixse\automlbenchmark\resources\benchmarks\test.yaml. An error occurred (403) when calling the HeadBucket operation: Forbidden Traceback (most recent call last): File "runbenchmark.py", line 113, in <module> bench.setup(automl.Benchmark.SetupMode[args.setup]) File "C:\Users\riooms\OneDrive-DeloitteO365D)\Thesis\Benchmark\fixse\automlbenchmark\automl\aws.py", line 107, in setup self.bucket = self._create_s3_bucket() File "C:\Users\riooms\OneDrive-DeloitteO365D)\Thesis\Benchmark\fixse\automlbenchmark\automl\aws.py", line 498, in _create_s3_bucket raise e File "C:\Users\riooms\OneDrive-DeloitteO365D)\Thesis\Benchmark\fixse\automlbenchmark\automl\aws.py", line 484, in _create_s3_bucket self.s3.meta.client.head_bucket(Bucket=bucket_name) File "C:\Users\riooms\OneDrive-DeloitteO365D)\Thesis\Benchmark\automlbenchmark\venv\lib\site-packages\botocore\client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "C:\Users\riooms\OneDrive-DeloitteO365D)\Thesis\Benchmark\automlbenchmark\venv\lib\site-packages\botocore\client.py", line 661, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (403) when calling the HeadBucket operation: Forbidden I guess that's probably due to the fact that the aws config file for windows is empty in the examples folder in this new branch

[sebhrusen]

@Stefanenrichard the ~/.config/automlbenchmark/config.yaml file is merged on top of the default one, so even if it were completely empty, it wouldn't hurt.

And the line numbers in your stack above match with master branch, but not with fix/22/aws_wrong_url_on_win branch, so I guess you're still using master. Please note that this fix branch also fixes the examples/aws/config.yaml: it was pointing to an old forked repo. So you please double-check the project_repository in the file you copied to your user dir.

[richooms]

@seb-h2o That's my bad, i made a mistake with git clone by pulling the master. I still get the same error though. The lines just differ:

(venv) C:\Users\riooms\OneDrive-DeloitteO365D)\Thesis\Benchmark\automlbenchmark-fix-22-aws_wrong_url_on_win>python runbenchmark.py constantpredictor test -m aws Runningconstantpredictorontestbenchmarks inawsmode. Loading frameworks definitions from C:\Users\riooms\OneDrive-DeloitteO365D)\Thesis\Benchmark\automlbenchmark-fix-22-aws_wrong_url_on_win\resources\frameworks.yaml. Loading benchmark definitions from C:\Users\riooms\OneDrive-DeloitteO365D)\Thesis\Benchmark\automlbenchmark-fix-22-aws_wrong_url_on_win\resources\benchmarks\test.yaml. An error occurred (403) when calling the HeadBucket operation: Forbidden Traceback (most recent call last): File "runbenchmark.py", line 113, in <module> bench.setup(automl.Benchmark.SetupMode[args.setup]) File "C:\Users\riooms\OneDrive-DeloitteO365D)\Thesis\Benchmark\automlbenchmark-fix-22-aws_wrong_url_on_win\automl\aws.py", line 108, in setup self.bucket = self._create_s3_bucket() File "C:\Users\riooms\OneDrive-DeloitteO365D)\Thesis\Benchmark\automlbenchmark-fix-22-aws_wrong_url_on_win\automl\aws.py", line 499, in _create_s3_bucket raise e File "C:\Users\riooms\OneDrive-DeloitteO365D)\Thesis\Benchmark\automlbenchmark-fix-22-aws_wrong_url_on_win\automl\aws.py", line 485, in _create_s3_bucket self.s3.meta.client.head_bucket(Bucket=bucket_name) File "C:\Users\riooms\OneDrive-DeloitteO365D)\Thesis\Benchmark\automlbenchmark-fix-22-aws_wrong_url_on_win\venv\lib\site-packages\botocore\client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "C:\Users\riooms\OneDrive-DeloitteO365D)\Thesis\Benchmark\automlbenchmark-fix-22-aws_wrong_url_on_win\venv\lib\site-packages\botocore\client.py", line 661, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (403) when calling the HeadBucket operation: Forbidden

[sebhrusen]

@Stefanenrichard something I don't understand:

• in your first original error, it was failing when trying to upload files to s3 from your EC2 instance, which means that not only the bucket already existed or was previously successfully created, but the ec2 instance was even successfully started.
• in your last error, it is failing much earlier, when creating the bucket itself, before file upload even starts.

Any other change (outside the fix branch) between those 2 errors? The last error looks like a AWS auth issue: I'm wondering if it was not trying to access the default bucket to which you definitely don't have access. Do you still have bucket: thesisbenchooms set in your config file? This name needs to be unique across S3.

I pushed some updates to the fix branch, it should improve logging. Please update your local branch, and if you don't mind, add the last logs/runbenchmark_xxxxxx.log in your next post (there's no AWS auth keys logged or anything, no worry: those are detected and handled directly by boto3).

[richooms]

Hi @seb-h2o, you were right. I messed something up with the versions and did not set everything up in a good manner last night. I'm sorry for that, as you are really helpful. My latest log is in here, there still is an error. I'm not sure if it is in the setup due to the missing SSH keys as it says AccessDenied or that it is due to some files missing, which I would not think is the case.

runbenchmark_20190619T083435_full.log

Checking on the case I found these links https://github.com/aws/aws-cli/issues/2408 https://stackoverflow.com/questions/38774798/accessdenied-for-listobjects-for-s3-bucket-when-permissions-are-s3

So its something in the configuration. Should I adapt this locally, or do you think it is something that should be adapted in this benchmark branch?

Furthermore, this is just a tiny little thing: when installing the app the first time an (empty) logs folder is missing. The user needs to create one, maybe you could add that to the instructions for the getting started instructions. should be in the setup section in my opinion.

[sebhrusen]

Hi @Stefanenrichard, I should thank you for discovering those bugs indeed. Basically, what happens is that you're using temporary s3 buckets (the benchmark app creates and deletes them on demand) but the IAM policy is not temporary, which is probably a use-case that was not tested. As you previously already managed to start an EC2 instance with the app, the IAM policy was then created, but now it doesn't match anymore the bucket it was created for, hence the AccessDenied (I noticed in the logs that there was no creation of new role policy). multiple ways to fix this:

1. use temporary IAM policy (configurable in your config.yaml: aws.iam.temporary: true.
2. update the git branch I created for this ticket (the one you're currently using): I pushed a fix there to slightly broaden the IAM policy in this case.

I would recommend the 2nd solution: the problem with new role policies is that they take a while to propagate to new EC2 instances, several minutes indeed, and the app will wait 6min by default before starting a new instance after creating a new policy... You will also have to force regeneration of the policy next time, this is done by using the --setup=force or -s force command line params.

To sum up:

> git pull
> python runbenchmark.py ....  -m aws -s force

Try not to use this -s force flag every time, otherwise it will always regenerate the roles.

[sebhrusen]

Furthermore, this is just a tiny little thing: when installing the app the first time an (empty) logs folder is missing. The user needs to create one, maybe you could add that to the instructions for the getting started instructions. should be in the setup section in my opinion.

Oh damn, you're right! Fixing this, it should be created on the fly, pretty sure it used to work.. Thanks!

[richooms]

Hi @seb-h2o,

You have done it! I it works :) (logs included as proof). This is amazing! Thank you very much :) If you need any more testing on windows for anything, I'm glad to help!

runbenchmark_20190619T120931_full.log

[sebhrusen]

Great! merging the fix branch then.

[sebhrusen]

All those fixes as now available on master and from v1.0.1