search

bytedance / bitsail

BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.
https://bytedance.github.io/bitsail/
Apache License 2.0
1.63k stars 335 forks source link

[Improve][Connector] Change `path_list` option setting using concating string with comma `,` to string list for FtpConnector #424

Open Jake-00 opened 1 year ago

Jake-00 commented 1 year ago

Description

For FtpConnector, currently we can specify directories to read by setting configuration json file using option path_list, but it can't specify several files' path to read.

Suggetion:

  1. currently path_list specify a string contains path list, it is better to change to specify a string list like below
  2. update documentation of FtpCoinnector 
    "reader": {
  "path_list": ["/data/json/upload1/test1.json", "/data/json/upload2/"]
}

BitSail Component or Code Module

BitSail Connector

Are you willing to submit PR?

Code of Conduct

Jake-00 commented 1 year ago

This issue maybe good for beginner.

hk-lrzy commented 1 year ago

@Jake-00 Thanks for your issue, you suggestion is we just not support directories right, we can use file path directly?

Jake-00 commented 1 year ago

@hk-lrzy To read directories already support, but to read multiple specified files do not support yet.

hk-lrzy commented 1 year ago

@hk-lrzy To read directories already support, but to read multiple specified files do not support yet.

I see, can you use same parameter to support? like we check the path is file or directory, if directory we support to list it and if it is file we support read it directly.

I don't want to add parameter because user need to know use which one, it's not necessary for user.

Jake-00 commented 1 year ago

Sounds more reasonable, shall I change the issue description?

hk-lrzy commented 1 year ago

Sounds more reasonable, shall I change the issue description?

Sure~

Jake-00 commented 1 year ago

When diving deeper in FtpConnector, it already supports to read specified files and directories. Issue description is wrong and I change the description of this issue. Could we add tag good-first-issue?