[Bug] fatal: remote error: upload-pack: not our ref

[canivel]

Hi, ECS task test-controller is failing with the error: Fetching submodule 3rdparty/secp256k1 fatal: remote error: upload-pack: not our ref fatal: the remote end hung up unexpectedly panic: runtime error: index out of range [-1]]

[HalosGhost]

@canivel thank you for reporting! A lot of us are surprisingly busy this week, so you might not get a response/resolution right away, but know that this is on our radar!

[canivel]

any news?

[HalosGhost]

Unfortunately “this week” turned into “this month”, but this needs to be addressed. I'll try to take a look at this personally in the next day or so, and I'm also going to chat with a few people more knowledgeable than I on this subject. ☺

[bvkin]

Hello @canivel, is your environment using the mit-dci version of the test controller, i.e., this one?

[HalosGhost]

(by the way, @bvkin is definitely one of those more-knowledgeable people!)

[canivel]

Hi bvkin I will check the version and revert back, but as far as I understand probably is using the last version, I did not changed anything on the deployment

[canivel]

@bvkin from the variables.tf i just tried again from scratch and same error: The error happen on the last stage of the Code pipeline : Deploy > Controller

[canivel]

hello? any news?

[HalosGhost]

Yes! We're working to reproduce this bug in our setup (we're standing up a full copy of the environment from-scratch).

[canivel]

Sorry to ask again but any luck?

[HalosGhost]

No worries and no problem. The short version, I believe, is that the test controller is a bit out-of-date compared to the transaction-processor and the terraform repo. I have a “release candidate” branch setup on the test controller which includes a lot of fixes/features/improvements that will hopefully solve most issues. I'd wonder if you can retarget your test_controller_github_repo_branch to point to rc/2022-Nov-14. Please let me know if that solves the issue for you or if it's still a problem. That double-check and testing would be incredibly valuable!

Note that the reason those changes haven't been pushed to trunk quite yet is because I'm still testing them to ensure everything works, so running into other issues may be possible. ☺

[canivel]

Awesome it's all running now, will test the components and let you know. The issue with the requirements cp to S3 still there, I commented the line 533 of main./tf and it compiled fine

• aws --endpoint-url $S3_INTERFACE_ENDPOINT s3 cp ./requirements.txt s3://$S3_BUCKET/test-controller-agent/ Thank you!

`

[Container] 2022/11/15 18:20:12 Running command aws --endpoint-url $S3_INTERFACE_ENDPOINT s3 cp ./requirements.txt s3://$S3_BUCKET/test-controller-agent/

108 |   109 | The user-provided path ./requirements.txt does not exist. 110 |   111 | [Container] 2022/11/15 18:20:12 Command did not exit successfully aws --endpoint-url $S3_INTERFACE_ENDPOINT s3 cp ./requirements.txt s3://$S3_BUCKET/test-controller-agent/ exit status 255 112 | [Container] 2022/11/15 18:20:12 Phase complete: BUILD State: FAILED 113 | [Container] 2022/11/15 18:20:12 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: aws --endpoint-url $S3_INTERFACE_ENDPOINT s3 cp ./requirements.txt s3://$S3_BUCKET/test-controller-agent/. Reason: exit status 255

`

[HalosGhost]

I've still not been able to reproduce this issue. You said you commented out main.tf:533? Can you link to the line? The place I'm looking doesn't seem to map to what you're referring to.

[canivel]

[HalosGhost]

clickable perma-link: https://github.com/mit-dci/terraform-aws-opencbdc-tctl/blob/9bf8815aec22d487d862bf4c51aa49fa1f5eef17/modules/test-controller-deploy/main.tf#L534

[HalosGhost]

@bvkin we didn't run into this issue during our testing; any idea what might have enabled us to get by unscathed?