[Bug]: DuckDB source fails build with Git LFS

[archiewood]

Describe the bug

If you make an evidence project with a large duckdb file (>100MB) and put it in git, then it has to be stored as a git LFS object, which is not a duckdb file, but a pointer file.

This will build locally, but will fail in some (all?) deployment environments. At least Evidence Cloud.

[ ! ] Error connecting to datasource [source_name]: IO Error: 
The file "/path/to/fs/sources/[source_name]/[your_db].duckdb" exists, but it is not a valid DuckDB database file!

This is likely because DuckDB is looking for a duckdb file, but instead it sees a git LFS pointer.

Steps to Reproduce

• Create a evidence project.
• Add a duckdb data source which is > git LFS limit, and commit using LFS object.
• Attempt to deploy to evidence cloud

Severity

serious, but I can work around it

Additional Information, or Workarounds

Store your data in a cloud database.

• Use motherduck.
• Use object storage eg S3.

[csjh]

I think the solution here would just be to pull the lfs object itself with git lfs pull --include="path/to/file, not sure if there's anything we can or should do on our end

[archiewood]

@Kanishk1802 did your workaround function as expected?

[Kanishk1802]

@archiewood the build is still running on cloud, waiting on it to finish to confirm it works

[Kanishk1802]

tentatively think it has worked as the 'npm run sources' step passed on cloud but I'll verify again once the deployment is completed

[Kanishk1802]

@archiewood my build succeeded, this can be listed as a workaround for this issue:

Enable storage of Git LFS objects in archive, for the repo powering build.