Closed nnamacha closed 3 months ago
Hi @nnamacha !
Thanks for raising this issue and thanks for contributing in making this project better!
Sorry to hear you have this kind of issue with flows. Flow metadata has a lot of treatment, especially around translation.
Do you have translations related to this flow definition in the repo (even not changed in the commit range you are targetting) ?
Could you send us the flow from the commit and the related translation files (anonymised if you can) so I can reproduce the issue on my side please ?
Flows.zip Find attached . Am not sure what you mean by translations related to the flow.
Hi @nnamacha
Thanks for the flows. I checked locally and there is no performance issue when the repo contains only them.
I think the performance issue you have is related to translations.
Because when a Flow is deployed, the related translations are dropped from Salesforce, so the plugin also add the translation related to the flow in the output and in the package.xml
to prevent that.
Could you also provide all the translations file with all their content from your repository please ?
Even if they do not contains flowDefinitions
they will be evaluated by the plugin, then they will count.
Based on your context I'll look at a way to optimize this treatment as much as possible.
In our environment we do not use translations .
Ok, so maybe your repository is huge ? Can I fork it and see for myself ?
Yes it is huge. Unfortunately it's a private repo and we cannot share it
Ok, that's a strong lead.
It should be way much faster with this release we are currently preparing.
Stay tuned
Hi @nnamacha,
We released a latest-rc
revision.
Could you install it and try with this new version please ?
Let us know the result please
After upgrading to v5.35.0 we are now getting this error
Hi @nnamacha
The new sgd version uses os.availableParallelism (only available since node v18.14
).
The node version you have is v18.12.1
.
I think this is the reason why you have this error, it is not available at your node version.
Sfdx cli recommends to us node Active LTS Version (v20.11.0
)
I think you can test 2 things:
v18.19
v20.11.0
And let us know please !
Noted
Am now getting
Interesting @nnamacha
Could you give me read access to the repository please (or a clone or an extract - you can send it by email, the email is on my github profile) ? I'm afraid I need to debug in order to investigate further.
@scolladon Unfortunately I can not share . Is there a way of enabling some king of logging and I can send you the log?
Hi @nnamacha, I think we can still do something else before having to go with a screenshare.
I have added a lot of traces in this PR You can install it locally following those steps. And then run sgd as usual.
To reapply the main sgd version just redo the normal installation via sfdx plugins:install
Please share the output as a file (it should be pretty huge so you may want to redirect sgd output to a file directly)
Am not sure if am missing
What is the branch the sfdx-git-delta project is on @nnamacha ?
fix/slice-issue
Could you run yarn && yarn pack
inside the sfdx-git-delta folder and then retry please ?
Ok, I have added more trace
Could you update the branch fix/slice-issue
locally, rebuild the plugin and then try again please ?
$ cd ~/sfdx_projects/sfdx-git-delta
$ git checkout fix/slice-issue
$ git pull
$ yarn
$ yarn pack
$ cd ~/salesforce-devops
$ sfdx sgd:source:delta --to "HEAD" --from $(git merge-base "HEAD" "origin/master") --output "." --api-version "59.0" --generate-delta > sgd_output
Weird, the issue seems to be linked to the way the repo is packed... Maybe the repository is very old on your machine.
Could you try with a fresh clone of your repository somewhere else please, just to see if it changes the output ? I suspect the issue comes from the repo state in your local. If it is the case we will either fix it or you could start from a fresh clone if all the work done is already in the remote.
But why is it if I downgrade to 5.34.0 I have no issues. It works besides when there are flows in the build it takes very long but eventually completes successfully.
Because in this version we are using another tool to communicate with git. The latest-rc version uses a new adapter more portable and performant
Ok let me try that
I asked my colleague to try and it worked with v5.35.0
niiiice 🎊
It means it is your local that needs maintenance.
You can either swap a new fresh clone
Or try to play with the git maintenance
command locally. I think it has a lot of tools to show errors and to understand/display what it is going to do before applying it. If your repository is huge and you have a lot of development with a lot of developers then maybe you could consider having a recurrent task to maintain the remote repo once in a while.
What about the performance with flow using the v5.35.0
?
Hi @nnamacha !
How it the performance with the new version ? Can we close this issue or is there anything else we can help with ?
Apologies for the last response, I wasn't available for a couple of days. I recloned the repo and still got the same error. The other developers are also getting the same . Find attached the details from one of my colleague who was successful.
Ok, could you try one more thing please @nnamacha ?
Maybe the repository is a bit messed up, I propose to "try to repair it". Here is how I propose to do it:
$ git clone <your-repo-uri> <somewhere>
$ cd <somewhere>
# we want to see if there issue found by git itself
$ git fsck --full
# repack everything properly (better than `git gc --aggressive` cf https://stackoverflow.com/questions/28720151/git-gc-aggressive-vs-git-repack)
$ git for-each-ref --format='delete %(refname)' refs/original | git update-ref --stdin
$ git filter-branch -f --index-filter "git rm -rf --cached --ignore-unmatch public/images/upload_xx/" -- --all
$ git filter-branch -f --index-filter "git rm -rf --cached --ignore-unmatch public/images/upload_xx/" --tag-name-filter cat -- --all
$ git reflog expire --expire=now --all
$ git repack -a -d -f --depth=50 --window=250
# prune dangling pack
$ git prune
# recheck
$ git fsck --full
# rerun sgd
$ sfdx sgd:source:delta --to "HEAD" --from $(git merge-base "HEAD" "origin/master") --output "." --api-version "59.0" --generate-delta
If it does not work then try git gc --aggressive --no-cruft --prune=now
and see if it is better.
Do not do those steps in your current repo, please clone and do that locally inside a new one. Let us know the result.
Getting same error
Sorry didn't run git gc --aggressive --no-cruft --prune=now . Running it now
Do you have better output ?
If none, I think the issue could reside in the underlying library we are using. I may have to patch it so it works with your use case, but doing so I need a way to reproduce the issue. Having access to your repository would be the easiest.
We are still getting it. Can we have a session where I can share my screen?
Yes, I'm available tomorrow afternoon (Paris time) My email is on my public github profile
Hi @nnamacha ! I hope you are well, it was nice to see you guys last time.
I come back to you to know if you managed to give me some kind of access to your repository so I can reproduce locally and debug the issue.
It seems to be an isolated case but I don't want other people to get the same issue as you so I really want to investigate and put effort into fixing this issue.
Please, give me access to a repo where I can reproduce (a fork or a subset where the issue occurs anyway or an isolated scenario where the issue is reproduced)
Apologies I wasn't available again for a couple of days. Am now requesting the access for you.
Have you received the email granting you access?
Nop, not yet @nnamacha
Have you been able to give me access @nnamacha ? It does not seems to be the case on my end
@nnamacha sorry to insist, could you give me access ?
It's ok, I had to follow up again for approval. It has been approved you should get the email now
That's great news
Is it possible to user your salesforce email address? sailpoint.connector.ConnectorException: Exception occurred. Error message - HTTP not ended OK. Response Code - 400 Error - Your organization does not allow collaboration with the domain of the user you're inviting. Contact your admin to allow collaboration with this domain.
Any update on this issue?
Issue verification check:
What is the problem?
When running git delta with changes involving FlowDefinitions the process is just hanging . When you remove the flows from the build the package.xml is created successfully
What is the parameter and the value you used with it?
Provide the command and the parameters you used sfdx sgd:source:delta --to "HEAD" --from $(git merge-base "HEAD" "origin/master") --otead.utput . --api-version 59.0 --generate-delta
What is the expected result?
Package.xml file generated
What is the actual result?
The process just hangs after generating the components before creating the package.xml files
Steps to reproduce
$ uname -v ; yarn -v ; node -v ; git --version ; sfdx --version ; sfdx plugins 2022-09-05 20:28 UTC 1.22.21 v18.12.1 git version 2.39.0.windows.1 @salesforce/cli/2.23.20 win32-x64 node-v18.12.1 @copado/copado-cli 3.1.0 dependencies-cli 2.0.1 sfdx-git-delta 5.34.0 sfdx-hardis 4.5.1 sfdx-plugin-source-read 1.2.0
Uninstalled JIT Plugins: community 3.0.10 custom-metadata 3.0.12 dev 2.1.3 devops-center 1.2.5 env 3.0.13 functions 1.22.11 signups 2.0.13 @salesforce/sfdx-plugin-lwc-test 1.1.1 @salesforce/sfdx-scanner 3.19.0