Cloud9, ExpiredToken mishandling, `XMLError: XML declaration allowed only at the start of the document from XML parser`

The expired token was getting injected in the result that the AWS SDK was returning. The doubled up xml failed in parsexml in XMLDict. This appears to be a kind of race condition on token refresh on AWS Cloud9. If I wait long enough between AWS calls, the next call has this error very consistently.

Maybe the solution is to split the response from the AWS SDK if the string <?xml version=\"1.0\" encoding=\"UTF-8\"?> shows up more than once? For now I am putting a dummy call before my real call with a try catch around it, just so the ExpiredToken, doesn't crash my process.

XMLError: XML declaration allowed only at the start of the document from XML parser (code: 64, line: 2)
Stacktrace:
[1] throw_xml_error
@ ~/.julia/packages/EzXML/ZNwhK/src/error.jl:87
[2] macro expansion
@ ~/.julia/packages/EzXML/ZNwhK/src/error.jl:52 [inlined]
[3] parsexml
@ ~/.julia/packages/EzXML/ZNwhK/src/document.jl:80
[4] parse_xml
@ ~/.julia/packages/XMLDict/vlQGP/src/XMLDict.jl:60 [inlined]
[5] _read
@ ~/.julia/packages/AWS/5goc1/src/utilities/response.jl:61
[6] #34
@ ~/.julia/packages/AWS/5goc1/src/utilities/response.jl:53 [inlined]
[7] _rewind
@ ~/.julia/packages/AWS/5goc1/src/utilities/response.jl:120
[8] parse
@ ~/.julia/packages/AWS/5goc1/src/utilities/response.jl:52 [inlined]
[9] parse (repeats 2 times)
@ ~/.julia/packages/AWS/5goc1/src/utilities/response.jl:58
[10] #parse#3
@ ~/.julia/packages/AWSS3/D1gQl/src/AWSS3.jl:78 [inlined]
[11] parse
@ ~/.julia/packages/AWSS3/D1gQl/src/AWSS3.jl:78
[12] _s3_exists_file
@ ~/.julia/packages/AWSS3/D1gQl/src/AWSS3.jl:231
[13] s3_exists_unversioned
@ ~/.julia/packages/AWSS3/D1gQl/src/AWSS3.jl:301 [inlined]
[14] #s3_exists#13
@ ~/.julia/packages/AWSS3/D1gQl/src/AWSS3.jl:316 [inlined]
[15] s3_exists
@ ~/.julia/packages/AWSS3/D1gQl/src/AWSS3.jl:313
[16] #s3_exists#14
@ ~/.julia/packages/AWSS3/D1gQl/src/AWSS3.jl:319
[17] s3_exists
@ ~/.julia/packages/AWSS3/D1gQl/src/AWSS3.jl:319

XMLDict.jl

 57 # Parse "xml" string into EzXML.Document object.
 58 
 59 function parse_xml(xml::AbstractString)
 60     @show xml  # ADDED THIS LINE
 61     doc = parsexml(xml)
 62     return wrap(doc, doc)
 63 end

Putting an xml dump in XMLDict showed me this (but without the whitespace):

<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<Error>
    <Code>ExpiredToken</Code>
    <Message>The provided token has expired.</Message>
    <Token-0>XXXXXXXXXXXX-REAL-TOKEN-XXXXXXXXXXX
    </Token-0>
    <RequestId>2T49JJB4PE1JJFTE</RequestId>
    <HostId>Q3rSq7HBBdAiax0RgYYYYYY</HostId>
</Error>

<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<ListBucketResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\">
    <Name>my-bucket-asdf</Name>
    <Prefix>my-s3-path</Prefix>
    <KeyCount>1</KeyCount>
    <MaxKeys>1</MaxKeys>
    <IsTruncated>false</IsTruncated>
    <Contents>
        <Key>my-s3-path</Key>
        <LastModified>2022-06-02T15:43:33.000Z</LastModified>
        <ETag>&quot;cb5ae17636e975f9bf71ddf5bc542075&quot;</ETag>
        <Size>3</Size>
        <StorageClass>STANDARD</StorageClass>
    </Contents>
</ListBucketResult>

AWS cli version included in Cloud9 on ubuntu 18.04

aws --version
aws-cli/1.23.2 Python/3.6.9 Linux/5.4.0-1075-aws botocore/1.25.2

Julia version

julia version 1.6.5

So we found a solution outside of julia for this...

https://aws.amazon.com/premiumsupport/knowledge-center/ec2-expired-token/

The steps we followed were to install an Instance IAMRole and not rely on the 5 minute or 15 minute token files that are defaulted in Cloud9.

Turn off the current credentials scheme (https://www.eksworkshop.com/020_prerequisites/workspaceiam/)

aws cloud9 update-environment  --environment-id $C9_PID --managed-credentials-action DISABLE
rm -vf ${HOME}/.aws/credentials

Create an instance IAM role...

Select trusted entity
    AWS service
Use case
    EC2
Add permissions # similar to developer IAM group
    (what you need for julia to perform thru AWS SDK)
Name
    Cloud9Instance # or whatever you want to call it

Go to the EC2 instance for the Cloud9 environment

Context: EC2 > Instance (for cloud9)

    Actions > Security > Modify IAM role
        Cloud9Instance

Verify the instance is getting the new credentials properly.

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/Cloud9Instance
aws configure list

Note that the credentials refresh once a day instead of every few minutes.

JuliaCloud / AWS.jl

Cloud9, ExpiredToken mishandling, `XMLError: XML declaration allowed only at the start of the document from XML parser` #556