Git delta hanging with an error with flows

nnamacha commented 7 months ago

Issue verification check:

[x] is the current repository fully deployable at the commit SHA provided with the 'from' parameter of the command?

What is the problem?

When running git delta with changes involving FlowDefinitions the process is just hanging . When you remove the flows from the build the package.xml is created successfully

What is the parameter and the value you used with it?

Provide the command and the parameters you used sfdx sgd:source:delta --to "HEAD" --from $(git merge-base "HEAD" "origin/master") --otead.utput . --api-version 59.0 --generate-delta

What is the expected result?

Package.xml file generated

What is the actual result?

The process just hangs after generating the components before creating the package.xml files

Steps to reproduce

$ uname -v ; yarn -v ; node -v ; git --version ; sfdx --version ; sfdx plugins 2022-09-05 20:28 UTC 1.22.21 v18.12.1 git version 2.39.0.windows.1 @salesforce/cli/2.23.20 win32-x64 node-v18.12.1 @copado/copado-cli 3.1.0 dependencies-cli 2.0.1 sfdx-git-delta 5.34.0 sfdx-hardis 4.5.1 sfdx-plugin-source-read 1.2.0

Uninstalled JIT Plugins: community 3.0.10 custom-metadata 3.0.12 dev 2.1.3 devops-center 1.2.5 env 3.0.13 functions 1.22.11 signups 2.0.13 @salesforce/sfdx-plugin-lwc-test 1.1.1 @salesforce/sfdx-scanner 3.19.0

nnamacha commented 7 months ago

scolladon commented 7 months ago

Hi @nnamacha !

Thanks for raising this issue and thanks for contributing in making this project better!

Sorry to hear you have this kind of issue with flows. Flow metadata has a lot of treatment, especially around translation.

Do you have translations related to this flow definition in the repo (even not changed in the commit range you are targetting) ?

Could you send us the flow from the commit and the related translation files (anonymised if you can) so I can reproduce the issue on my side please ?

nnamacha commented 7 months ago

Flows.zip Find attached . Am not sure what you mean by translations related to the flow.

scolladon commented 7 months ago

Hi @nnamacha

Thanks for the flows. I checked locally and there is no performance issue when the repo contains only them.

I think the performance issue you have is related to translations. Because when a Flow is deployed, the related translations are dropped from Salesforce, so the plugin also add the translation related to the flow in the output and in the package.xml to prevent that.

Could you also provide all the translations file with all their content from your repository please ? Even if they do not contains flowDefinitions they will be evaluated by the plugin, then they will count. Based on your context I'll look at a way to optimize this treatment as much as possible.

nnamacha commented 7 months ago

In our environment we do not use translations .

scolladon commented 7 months ago

Ok, so maybe your repository is huge ? Can I fork it and see for myself ?

nnamacha commented 7 months ago

Yes it is huge. Unfortunately it's a private repo and we cannot share it

scolladon commented 7 months ago

Ok, that's a strong lead.

It should be way much faster with this release we are currently preparing.

Stay tuned

scolladon commented 7 months ago

Hi @nnamacha,

We released a latest-rc revision. Could you install it and try with this new version please ? Let us know the result please

nnamacha commented 7 months ago

After upgrading to v5.35.0 we are now getting this error

nnamacha commented 7 months ago

scolladon commented 7 months ago

Hi @nnamacha

The new sgd version uses os.availableParallelism (only available since node v18.14). The node version you have is v18.12.1. I think this is the reason why you have this error, it is not available at your node version.

Sfdx cli recommends to us node Active LTS Version (v20.11.0)

I think you can test 2 things:

either upgrade your current node maintenance LTS Version to latest => v18.19
either upgrade your current node to the active LTS Version => v20.11.0

And let us know please !

nnamacha commented 7 months ago

Noted

nnamacha commented 7 months ago

Am now getting

nnamacha commented 7 months ago

scolladon commented 7 months ago

Interesting @nnamacha

Could you give me read access to the repository please (or a clone or an extract - you can send it by email, the email is on my github profile) ? I'm afraid I need to debug in order to investigate further.

nnamacha commented 7 months ago

@scolladon Unfortunately I can not share . Is there a way of enabling some king of logging and I can send you the log?

scolladon commented 7 months ago

Yes there is a way... It is a bit more hands on and would require interaction I guess.

Would you be able to connect to a screen share so I can guide you ? You will have to install the development version locally following those steps first with this PR I am available today whenever you want

scolladon commented 7 months ago

Hi @nnamacha, I think we can still do something else before having to go with a screenshare.

I have added a lot of traces in this PR You can install it locally following those steps. And then run sgd as usual.

To reapply the main sgd version just redo the normal installation via sfdx plugins:install

Please share the output as a file (it should be pretty huge so you may want to redirect sgd output to a file directly)

nnamacha commented 7 months ago

Am not sure if am missing

scolladon commented 7 months ago

What is the branch the sfdx-git-delta project is on @nnamacha ?

nnamacha commented 7 months ago

fix/slice-issue

scolladon commented 7 months ago

Could you run yarn && yarn pack inside the sfdx-git-delta folder and then retry please ?

nnamacha commented 7 months ago

scolladon commented 7 months ago

Ok, I have added more trace Could you update the branch fix/slice-issue locally, rebuild the plugin and then try again please ?

$ cd ~/sfdx_projects/sfdx-git-delta
$ git checkout fix/slice-issue
$ git pull
$ yarn
$ yarn pack
$ cd ~/salesforce-devops
$ sfdx sgd:source:delta --to "HEAD" --from $(git merge-base "HEAD" "origin/master") --output "." --api-version "59.0" --generate-delta > sgd_output

nnamacha commented 7 months ago

sgd_output.log

scolladon commented 7 months ago

Weird, the issue seems to be linked to the way the repo is packed... Maybe the repository is very old on your machine.

Could you try with a fresh clone of your repository somewhere else please, just to see if it changes the output ? I suspect the issue comes from the repo state in your local. If it is the case we will either fix it or you could start from a fresh clone if all the work done is already in the remote.

nnamacha commented 7 months ago

But why is it if I downgrade to 5.34.0 I have no issues. It works besides when there are flows in the build it takes very long but eventually completes successfully.

scolladon commented 7 months ago

Because in this version we are using another tool to communicate with git. The latest-rc version uses a new adapter more portable and performant

nnamacha commented 7 months ago

Ok let me try that

nnamacha commented 7 months ago

I asked my colleague to try and it worked with v5.35.0

scolladon commented 7 months ago

niiiice 🎊

It means it is your local that needs maintenance. You can either swap a new fresh clone Or try to play with the git maintenance command locally. I think it has a lot of tools to show errors and to understand/display what it is going to do before applying it. If your repository is huge and you have a lot of development with a lot of developers then maybe you could consider having a recurrent task to maintain the remote repo once in a while.

What about the performance with flow using the v5.35.0 ?

scolladon commented 7 months ago

Hi @nnamacha !

How it the performance with the new version ? Can we close this issue or is there anything else we can help with ?

nnamacha commented 7 months ago

Apologies for the last response, I wasn't available for a couple of days. I recloned the repo and still got the same error. The other developers are also getting the same . Find attached the details from one of my colleague who was successful.

scolladon commented 7 months ago

Ok, could you try one more thing please @nnamacha ?

Maybe the repository is a bit messed up, I propose to "try to repair it". Here is how I propose to do it:

$ git clone <your-repo-uri> <somewhere>
$ cd <somewhere>
# we want to see if there issue found by git itself
$ git fsck --full 
# repack everything properly (better than `git gc --aggressive` cf https://stackoverflow.com/questions/28720151/git-gc-aggressive-vs-git-repack)
$ git for-each-ref --format='delete %(refname)' refs/original | git update-ref --stdin
$ git filter-branch -f --index-filter "git rm -rf --cached --ignore-unmatch public/images/upload_xx/" -- --all
$ git filter-branch -f --index-filter "git rm -rf --cached --ignore-unmatch public/images/upload_xx/" --tag-name-filter cat -- --all
$ git reflog expire --expire=now --all
$ git repack -a -d -f --depth=50 --window=250
# prune dangling pack
$ git prune
# recheck
$ git fsck --full
# rerun sgd
$ sfdx sgd:source:delta --to "HEAD" --from $(git merge-base "HEAD" "origin/master") --output "." --api-version "59.0" --generate-delta

If it does not work then try git gc --aggressive --no-cruft --prune=now and see if it is better.

Do not do those steps in your current repo, please clone and do that locally inside a new one. Let us know the result.

nnamacha commented 7 months ago

Getting same error

nnamacha commented 7 months ago

Sorry didn't run git gc --aggressive --no-cruft --prune=now . Running it now

scolladon commented 7 months ago

Do you have better output ?

If none, I think the issue could reside in the underlying library we are using. I may have to patch it so it works with your use case, but doing so I need a way to reproduce the issue. Having access to your repository would be the easiest.

nnamacha commented 7 months ago

We are still getting it. Can we have a session where I can share my screen?

scolladon commented 7 months ago

Yes, I'm available tomorrow afternoon (Paris time) My email is on my public github profile

scolladon commented 6 months ago

Hi @nnamacha ! I hope you are well, it was nice to see you guys last time.

I come back to you to know if you managed to give me some kind of access to your repository so I can reproduce locally and debug the issue.

It seems to be an isolated case but I don't want other people to get the same issue as you so I really want to investigate and put effort into fixing this issue.

Please, give me access to a repo where I can reproduce (a fork or a subset where the issue occurs anyway or an isolated scenario where the issue is reproduced)

nnamacha commented 6 months ago

Apologies I wasn't available again for a couple of days. Am now requesting the access for you.

nnamacha commented 6 months ago

Have you received the email granting you access?

scolladon commented 6 months ago

Nop, not yet @nnamacha

scolladon commented 6 months ago

Have you been able to give me access @nnamacha ? It does not seems to be the case on my end

scolladon commented 6 months ago

@nnamacha sorry to insist, could you give me access ?

nnamacha commented 6 months ago

It's ok, I had to follow up again for approval. It has been approved you should get the email now

scolladon commented 6 months ago

That's great news

nnamacha commented 6 months ago

Is it possible to user your salesforce email address? sailpoint.connector.ConnectorException: Exception occurred. Error message - HTTP not ended OK. Response Code - 400 Error - Your organization does not allow collaboration with the domain of the user you're inviting. Contact your admin to allow collaboration with this domain.

nnamacha commented 5 months ago

Any update on this issue?

scolladon / sfdx-git-delta