Issues with incremental builds and disk space due to SourceRootMapping hashing bug

[ppanyukov]

The builds we are running always do new clone of repos. On our system this is due differences in hash code calculations between plugins/build/sourceMappings.js and scm/lib/scmprovider.js.

The result is:

• agent thinks that the entire repository needs to be cloned into new _work/NN directory.
• incremental builds obviously don't work
• agent runs out of disk space very quickly as there is a new clone for every build

The code in question:

plugins/build/sourceMappings.js:: https://github.com/Microsoft/vso-agent/blob/master/src/agent/plugins/build/sourceMappings.ts#L139

        var hashInput = expectedMap.collectionId + ':' + expectedMap.definitionId + ':' + endpoint.url;
        var hashProvider = crypto.createHash("sha256");
        hashProvider.update(hashInput, 'utf8');
        var hash = hashProvider.digest('hex');

scm/lib/scmprovider.js: : https://github.com/Microsoft/vso-agent/blob/master/src/agent/scm/lib/scmprovider.ts#L37

    public getHashKey() {
        var hash = null;

        if (this.endpoint.url) {
            var hashProvider = crypto.createHash("sha256");
            hashProvider.update(this.endpoint.url, 'utf8');
            hash = hashProvider.digest('hex');
        }

        return hash;
    }

I have confirmed that changing plugins/build/sourceMappings.js to only use endpoint.url for hashing resolves the issue once the _work/SourceRootMapping are deleted (which forces to created them).

This behaviour is a bit odd because I'm almost certain everything used to work correctly in the past. Not 100% sure though.

[ericsciple]

@ppanyukov Are you on visualstudio.com or using on-premises TFS?

[ppanyukov]

@ericsciple on visualstudio.com, using git repos.

[ericsciple]

Great! Can you try the new vsts-agent that is deprecating the vso-agent?

https://github.com/Microsoft/vsts-agent

[ppanyukov]

That is certainly planned, but at the moment we are stuck with this one for the moment. We have quite sophisticated setup, not as easy to swap things out .

[ppanyukov]

hmm, I'm no longer able to reproduce this. Neither with 0.4.5 version or with the latest 0.6.5. Very odd. Maybe something went corrupt at some point and that's what caused the issues. I have the logs _diag logs if anyone is interested. (still don't understand why we do hashing differently though).

Probably can close for now.

[bryanmacfarlane]

OK - it's also fixed in the replacement agent. You should switch over https://github.com/Microsoft/vsts-agent