pub get fails with polkadart git dependency: git lfs error

[clangenb]

Our CI fails to run melos bootstrap in our own flutter app when we add polkadart as a dependency. Locally, I could sometimes reproduce it, but I couldn't precisely figure out when it happens and when not. Do you have any idea? Note: I have overridden the polkadart dependencies to use our fork as a git depepency.

Below log is taken from this GitHub Acions run: https://github.com/encointer/encointer-wallet-flutter/actions/runs/6455192809/job/17522235609

melos bootstrap
  └> /Users/runner/work/encointer-wallet-flutter/encointer-wallet-flutter

Running "/Users/runner/work/encointer-wallet-flutter/encointer-wallet-flutter/.flutter/bin/flutter pub get" in workspace packages...
[encointer_wallet]: Git error. Command: `git clone /Users/runner/.pub-cache/git/cache/polkadart-4b9814cb663a50a9fa1ffa959ad565e5a4fd2e44 /Users/runner/.pub-cache/git/polkadart-500a3824beca7589b152351192ccc6cb6fbdc1df`
[encointer_wallet]: stdout: 
[encointer_wallet]: stderr: Cloning into '/Users/runner/.pub-cache/git/polkadart-500a3824beca7589b152351192ccc6cb6fbdc1df'...
[encointer_wallet]: done.
[encointer_wallet]: Downloading chain/kusama/blocks.jsonl (472 MB)
[encointer_wallet]: Error downloading object: chain/kusama/blocks.jsonl (b15cbcd): Smudge error: Error downloading chain/kusama/blocks.jsonl (b15cbcd17f218021932e663be0a5b1f21d673105fa8085bd9f3d9f52dbffef0c): error transferring "b15cbcd17f218021932e663be0a5b1f21d673105fa8085bd9f3d9f52dbffef0c": [0] remote missing object b15cbcd17f218021932e663be0a5b1f21d673105fa8085bd9f3d9f52dbffef0c
[encointer_wallet]: 
[encointer_wallet]: Errors logged to '/Users/runner/.pub-cache/git/polkadart-500a3824beca7589b152351192ccc6cb6fbdc1df/.git/lfs/logs/20231009T101236.406125.log'.
[encointer_wallet]: Use `git lfs logs last` to view the log.
[encointer_wallet]: error: external filter 'git-lfs filter-process' failed
[encointer_wallet]: fatal: chain/kusama/blocks.jsonl: smudge filter lfs failed
[encointer_wallet]: warning: Clone succeeded, but checkout failed.
[encointer_wallet]: You can inspect what was checked out with 'git status'
[encointer_wallet]: and retry with 'git restore --source=HEAD :/'
[encointer_wallet]: 
[encointer_wallet]: exit code: 128
  - encointer_wallet
       └> Failed to install.
    └> app
Git error. Command: `git clone /Users/runner/.pub-cache/git/cache/polkadart-4b9814cb663a50a9fa1ffa959ad565e5a4fd2e44 /Users/runner/.pub-cache/git/polkadart-500a3824beca7589b152351192ccc6cb6fbdc1df`

stdout: 
Resolving dependencies...
stderr: Cloning into '/Users/runner/.pub-cache/git/polkadart-500a3824beca7589b152351192ccc6cb6fbdc1df'...
done.
Downloading chain/kusama/blocks.jsonl (472 MB)
Error downloading object: chain/kusama/blocks.jsonl (b15cbcd): Smudge error: Error downloading chain/kusama/blocks.jsonl (b15cbcd17f218021932e663be0a5b1f21d673105fa8085bd9f3d9f52dbffef0c): error transferring "b15cbcd17f218021932e663be0a5b1f21d673105fa8085bd9f3d9f52dbffef0c": [0] remote missing object b15cbcd17f218021932e663be0a5b1f21d673105fa8085bd9f3d9f52dbffef0c
Errors logged to '/Users/runner/.pub-cache/git/polkadart-500a3824beca7589b152351192ccc6cb6fbdc1df/.git/lfs/logs/20231009T101236.406125.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: chain/kusama/blocks.jsonl: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'
exit code: 128
BootstrapException: Failed to install.: encointer_wallet at /Users/runner/work/encointer-wallet-flutter/encointer-wallet-flutter/app.
Error: Process completed with exit code 1.

[leonardocustodio]

I believe you can just add lfs: true in the checkout actions. Did you try that? https://github.com/actions/checkout

[clangenb]

Thanks for hint. Unfortunately, the problem is not when I check out the repository, it is when melos runs pub get. However, it seems that this a long-lasting issue: dart can't handle when a git-dependency contains lfs-files: https://github.com/dart-lang/pub/issues/1433.

So the naive question here would be: As I doubt that this will be fixed soon, is there any option that we can change the paradigm here?

[clangenb]

So for now, I could work around it if our fork hasn't the lfs stuff on the main branch (needs to be main, as pub get fails in the clone stage). So no need to hurry to fix that because I see that lfs is the simplest solution for now and regular dependents that fetch it polkadart from pub.dev are not affected anyhow. I will close the issue.

[leonardocustodio]

@clangenb can you try with this action we use in our CI, since we don't get any error when running our CI, though I understand it may not be the same behavior as this is our own project and not a dependency

https://github.com/leonardocustodio/polkadart/blob/main/.github/workflows/tests.yml

[clangenb]

Yes, this was my first try actually, but it doesn't work - it would work possibly if we checked in the pub-cache :P. The description in https://github.com/dart-lang/pub/issues/1433 even suggests cloning the dependency in question and copying the lfs directory into the pub-cache directory. If the lfs files are already there when pub get is run, everything is fine. This is also why it worked on my machine, as I added the lfs into the cache before I switched to the git dependencies.

With the above info I concluded then, I don't want to invest effort into a workaround for a broken framework, and will accept that the tests can only be run on a different branch, which is very easily maintainable.

[leonardocustodio]

I see, thanks for the information, when I've some free time I will remove the files from LFS, we can split them up easily so we don't need to use LFS to store them.

[clangenb]

Haha, as you will tackle the problem eventually, I will reopen the issue. Thanks a lot! :)

[leonardocustodio]

Closing as it will be solved with #340