std.filebase64sha256 behavior possibly incorrect

gbirman commented 8 months ago

What happened?

I'm noticing some strange behavior with std.filebase64sha256 where it doesn't seem to be reading the file contents properly, or at least outputs a completely different value than expected. Please see below.

Example

To illustrate the differences I created a file named text with the contents hi. Here are three different ways of setting the hash.

std
    .filebase64sha256({
      input: 'test',
    })
    .then((f) => console.log(`pulumi file hash: ${f.result}`));

std
    .base64sha256({
      input: 'hi',
    })
    .then((f) => console.log(`pulumi content hash: ${f.result}`));

const hash = crypto.createHash('sha256');
const fileBuffer = fs.readFileSync('test');
hash.update(fileBuffer);
const sourceCodeHash = hash.digest('base64');
console.log(`nodejs hash: ${sourceCodeHash}`);

Here is the output:

pulumi file hash: OGY0MzQzNDY2NDhmNmI5NmRmODlkZGE5MDFjNTE3NmIxMGE2ZDgzOTYxZGQzYzFhYzg4YjU5YjJkYzMyN2FhNA==
pulumi content hash: j0NDRmSPa5bfid2pAcUXaxCm2Dlh3TwayItZstwyeqQ=
nodejs hash: j0NDRmSPa5bfid2pAcUXaxCm2Dlh3TwayItZstwyeqQ=

As you can see the file hash is different from the content/nodejs hash, which are equivalent. The file hash is also outputting 88 characters instead of the expected 44 characters for a base64 encoded sha256.

Output of `pulumi about`

CLI
Version      3.111.1
Go Version   go1.22.1
Go Compiler  gc

Plugins
NAME     VERSION
archive  0.0.4
aws      6.27.0
awsx     2.6.0
docker   4.5.3
docker   3.6.1
nodejs   unknown
std      1.6.2

Host
OS       darwin
Version  14.1.1
Arch     arm64

This project is written in nodejs: executable='/Users/gabrielbirman/Library/Caches/fnm_multishells/45752_1711477108570/bin/node' version='v20.6.1'

Current Stack: macro-inc/link-sharing/dev

TYPE                                                    URN
pulumi:pulumi:Stack                                     urn:pulumi:dev::link-sharing::pulumi:pulumi:Stack::link-sharing-dev
pulumi:providers:aws                                    urn:pulumi:dev::link-sharing::pulumi:providers:aws::default_6_27_0
aws:iam/role:Role                                       urn:pulumi:dev::link-sharing::aws:iam/role:Role::api-gateway-rest-authorization-lambda-role-dev
aws:apigateway/restApi:RestApi                          urn:pulumi:dev::link-sharing::aws:apigateway/restApi:RestApi::get-presigned-url-rest-api-dev
pulumi:providers:archive                                urn:pulumi:dev::link-sharing::pulumi:providers:archive::default_0_0_4
aws:iam/role:Role                                       urn:pulumi:dev::link-sharing::aws:iam/role:Role::get-presigned-cloudfront-url-lambda-role-dev
aws:acm/certificate:Certificate                         urn:pulumi:dev::link-sharing::aws:acm/certificate:Certificate::get-presigned-url-rest-api-domain-cert-dev
aws:s3/bucket:Bucket                                    urn:pulumi:dev::link-sharing::aws:s3/bucket:Bucket::link-sharing-demo-file-storage-dev
pulumi:providers:std                                    urn:pulumi:dev::link-sharing::pulumi:providers:std::default_1_6_2
aws:dynamodb/table:Table                                urn:pulumi:dev::link-sharing::aws:dynamodb/table:Table::linksharing-demo-document-mapping-table-dev
aws:apigateway/resource:Resource                        urn:pulumi:dev::link-sharing::aws:apigateway/resource:Resource::get-presigned-url-rest-api-documents-resource-dev
aws:lambda/function:Function                            urn:pulumi:dev::link-sharing::aws:lambda/function:Function::api-gateway-rest-authorization-lambda-dev
aws:route53/record:Record                               urn:pulumi:dev::link-sharing::aws:route53/record:Record::get-presigned-url-rest-api-domain-cert-validation-dev
aws:iam/policy:Policy                                   urn:pulumi:dev::link-sharing::aws:iam/policy:Policy::linksharing-demo-document-mapping-table-read-access-policy-dev
aws:apigateway/resource:Resource                        urn:pulumi:dev::link-sharing::aws:apigateway/resource:Resource::get-presigned-url-rest-api-documents-id-resource-dev
aws:apigateway/authorizer:Authorizer                    urn:pulumi:dev::link-sharing::aws:apigateway/authorizer:Authorizer::get-presigned-url-rest-api-authorizer-dev
aws:apigateway/resource:Resource                        urn:pulumi:dev::link-sharing::aws:apigateway/resource:Resource::get-presigned-url-rest-api-location-resource-dev
aws:apigateway/domainName:DomainName                    urn:pulumi:dev::link-sharing::aws:apigateway/domainName:DomainName::get-presigned-url-rest-api-domain-name-dev
aws:apigateway/resource:Resource                        urn:pulumi:dev::link-sharing::aws:apigateway/resource:Resource::get-presigned-url-rest-api-permissions-resource-dev
aws:lambda/permission:Permission                        urn:pulumi:dev::link-sharing::aws:lambda/permission:Permission::get-presigned-url-rest-api-authorizer-get-invoke-permissions-dev
aws:apigateway/method:Method                            urn:pulumi:dev::link-sharing::aws:apigateway/method:Method::get-presigned-url-rest-api-permissions-get-method-dev
aws:apigateway/method:Method                            urn:pulumi:dev::link-sharing::aws:apigateway/method:Method::get-presigned-url-rest-api-location-options-method-dev
aws:apigateway/method:Method                            urn:pulumi:dev::link-sharing::aws:apigateway/method:Method::get-presigned-url-rest-api-permissions-options-method-dev
aws:apigateway/method:Method                            urn:pulumi:dev::link-sharing::aws:apigateway/method:Method::get-presigned-url-rest-api-location-get-method-dev
aws:route53/record:Record                               urn:pulumi:dev::link-sharing::aws:route53/record:Record::get-presigned-url-rest-api-domain-record-dev
aws:apigateway/methodResponse:MethodResponse            urn:pulumi:dev::link-sharing::aws:apigateway/methodResponse:MethodResponse::get-presigned-url-rest-api-permissions-get-method-response-dev
aws:apigateway/integration:Integration                  urn:pulumi:dev::link-sharing::aws:apigateway/integration:Integration::get-presigned-url-rest-api-location-options-integration-dev
aws:apigateway/integration:Integration                  urn:pulumi:dev::link-sharing::aws:apigateway/integration:Integration::get-presigned-url-rest-api-permissions-options-integration-dev
aws:apigateway/methodResponse:MethodResponse            urn:pulumi:dev::link-sharing::aws:apigateway/methodResponse:MethodResponse::get-presigned-url-rest-api-location-get-method-response-dev
aws:apigateway/integrationResponse:IntegrationResponse  urn:pulumi:dev::link-sharing::aws:apigateway/integrationResponse:IntegrationResponse::get-presigned-url-rest-api-location-options-integration-response-dev
aws:apigateway/methodResponse:MethodResponse            urn:pulumi:dev::link-sharing::aws:apigateway/methodResponse:MethodResponse::get-presigned-url-rest-api-location-options-method-response-dev
aws:apigateway/methodResponse:MethodResponse            urn:pulumi:dev::link-sharing::aws:apigateway/methodResponse:MethodResponse::get-presigned-url-rest-api-permissions-options-method-response-dev
aws:apigateway/integrationResponse:IntegrationResponse  urn:pulumi:dev::link-sharing::aws:apigateway/integrationResponse:IntegrationResponse::get-presigned-url-rest-api-permissions-options-integration-response-dev
aws:lambda/function:Function                            urn:pulumi:dev::link-sharing::aws:lambda/function:Function::get-presigned-cloudfront-url-lambda-dev
aws:apigateway/integration:Integration                  urn:pulumi:dev::link-sharing::aws:apigateway/integration:Integration::get-presigned-url-rest-api-location-get-integration-dev
aws:lambda/permission:Permission                        urn:pulumi:dev::link-sharing::aws:lambda/permission:Permission::get-presigned-url-rest-api-permissions-get-invoke-permissions-dev
aws:apigateway/integration:Integration                  urn:pulumi:dev::link-sharing::aws:apigateway/integration:Integration::get-presigned-url-rest-api-permissions-get-integration-dev
aws:lambda/permission:Permission                        urn:pulumi:dev::link-sharing::aws:lambda/permission:Permission::get-presigned-url-rest-api-location-get-invoke-permissions-dev
aws:apigateway/deployment:Deployment                    urn:pulumi:dev::link-sharing::aws:apigateway/deployment:Deployment::get-presigned-url-rest-api-deployment-dev
aws:apigateway/stage:Stage                              urn:pulumi:dev::link-sharing::aws:apigateway/stage:Stage::get-presigned-url-rest-api-stage-dev
aws:apigateway/basePathMapping:BasePathMapping          urn:pulumi:dev::link-sharing::aws:apigateway/basePathMapping:BasePathMapping::get-presigned-url-rest-api-mapping-dev
aws:apigateway/methodSettings:MethodSettings            urn:pulumi:dev::link-sharing::aws:apigateway/methodSettings:MethodSettings::get-presigned-url-rest-api-location-method-settings-dev

Found no pending operations associated with macro-inc/dev

Backend
Name           pulumi.com
URL            https://app.pulumi.com/gbirman
User           gbirman
Organizations  gbirman, macro-inc
Token type     personal

Dependencies:
NAME             VERSION
@pulumi/archive  0.0.4
@pulumi/aws      6.27.0
@pulumi/awsx     2.6.0
@pulumi/pulumi   3.111.1
@pulumi/std      1.6.2
@types/node      18.19.26
ts-node          10.9.2

Pulumi locates its logs in /var/folders/jh/w2d9x6j57_d3sn5n5l957yg40000gn/T/ by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

Zaid-Ajaj commented 8 months ago

Hi @gbirman thanks for filing the issue! This is a bug in the std provider. I'll move the the issue to the provider repo

Zaid-Ajaj commented 8 months ago

std.filebase64sha256 should have the same behaviour from terraform and this currently is not the case. To repro:

touch test.txt with content hi
terraform console to start up an interactive session
filebase64sha256("test.txt") will return j0NDRmSPa5bfid2pAcUXaxCm2Dlh3TwayItZstwyeqQ=

tgummerer commented 8 months ago

If you have a look at the values that are base64 encoded it becomes obvious what's happening.

$ echo "OGY0MzQzNDY2NDhmNmI5NmRmODlkZGE5MDFjNTE3NmIxMGE2ZDgzOTYxZGQzYzFhYzg4YjU5YjJkYzMyN2FhNA==" | base64 -d
8f434346648f6b96df89dda901c5176b10a6d83961dd3c1ac88b59b2dc327aa4

$ echo "j0NDRmSPa5bfid2pAcUXaxCm2Dlh3TwayItZstwyeqQ=" | base64 -d | xxd -p   
8f434346648f6b96df89dda901c5176b10a6d83961dd3c1ac88b59b2dc32
7aa4

So awkwardly filebase64sha256 seems to base64 encode the hex value of the hash, while the others base64 encode the binary value. Probably a bit of a tricky thing to fix as well because of backwards compatibility.

pulumi / pulumi-std