Universal SQL: Prevent attempting to read the contents of files over a maximum size

[ItsMeBrianD]

Not sure how we want to handle this; but I am unable to open a duckdb file that is 1.8GB in size because the plugin-connector is attempting to read the file as a string

[csjh]

Where/why is it being read?

[ItsMeBrianD]

By default all files in a sources directory that are not connection.yaml or connection.options.yaml are sent to the connector as potential source files; this includes reading the file contents to a string, along with the filepath

Because the duckdb file is not being skipped, it is being treated the same way

[csjh]

Are file contents read anywhere except in db adapters? Could just pass adapters an object along the lines of

{
    filepath: filepath,
    get contents() {
        return fs.readFileSync(filepath)
    }
}

could be fs.readFile but it's weird to require an await obj.contents

[ItsMeBrianD]

They shouldn't be, that may be a better approach. I'm not sure if doing that sync is a good move though if we want to parallelize data source execution

[ItsMeBrianD]

Agree that requiring the await isn't ideal

[archiewood]

Could you ignore certain binary extensions? (eg .duckdb .db .sqlite)

Or is this just not flexible in case someone drops in an big .exe or something else random

[ItsMeBrianD]

I've implemented Cormick's solution temporarily in the latest prerelease; we'll see if it has large implications on source building performance

[ItsMeBrianD]

I think a good way to approach this is to simply not load the file contents of large files; if the adapter wants to still handle those files, then the option will be there, the adapter will just have to load them itself

[csjh]

What are we going to consider a large file? >1MB?

[ItsMeBrianD]

I think this is currently implemented to an extent; using 100MB as the "large file" delimiter

[csjh]

Fixed by #1064