airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.51k stars 4k forks source link

Unclear documentation regarding local file source connector #45645

Open Joly0 opened 1 day ago

Joly0 commented 1 day ago

What happened?

Hey, i have just installed airbyte using the abctl command on a windows vm. I want to add a local folder so i can add that as a source for local files. The problem here is, that the documentation is talking about configs in the .env file, but that is when using the docker-compose approach which, according to the documentation, is the old way to do it and the new and recommended way is abctl instead. But i have not found any way using the abctl approach to add a local folder as a source.

What did you expect to happen?

A clear and concise documentation how to add a local source folder to airbyte using abctl

Abctl Version

```console $ abctl version version: v0.15.1 ```

Docker Version

```console $ docker version Client: Version: 27.2.0 API version: 1.47 Go version: go1.21.13 Git commit: 3ab4256 Built: Tue Aug 27 14:17:17 2024 OS/Arch: windows/amd64 Context: desktop-linux Server: Docker Desktop 4.34.1 (166053) Engine: Version: 27.2.0 API version: 1.47 (minimum version 1.24) Go version: go1.21.13 Git commit: 3ab5c7d Built: Tue Aug 27 14:15:15 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.7.20 GitCommit: 8fc6bcff51318944179630522a095cc9dbf9f353 runc: Version: 1.1.13 GitCommit: v1.1.13-0-g58aa920 docker-init: Version: 0.19.0 GitCommit: de40ad0 ```

OS Version

```console # On Linux: $ cat /etc/os-release # paste output here # On Mac: $ uname -a # paste output here # On Windows: C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture Invalid GET Expression. The correct command is: wmic os get Caption,Version,BuildNumber,OSArchitecture Output is: BuildNumber Caption OSArchitecture Version 20348 Microsoft Windows Server 2022 Standard 64-bit 10.0.20348 ```
techieshark commented 1 day ago

I had this issue as well (unclear how to access the CSV files or if even possible); I reported in Slack with some discussion there - https://airbytehq.slack.com/archives/C021JANJ6TY/p1726562874434619

Another user reported that the "files are written on destination connector pod file system, but data are gone when synchronization is done"

Joly0 commented 1 day ago

Thanks @techieshark i am trying to get into the slack, but it keeps complaining that i need an invitation.

techieshark commented 14 hours ago

@Joly0 I suspect you got in but just in case, here's where to get info about airbyte slack https://airbyte.com/community/community