wildcard bug when matched file count is 1

porscheme commented 2 years ago

Importing files with wildcard in the path failed. It works okay, with exact file name.

Below is the error I'm getting

C:\nebulaimporter>docker run --rm -it --add-host host.docker.internal:host-gateway --network host -v C:\coldstart\pending_vertices\test.vertices.yaml:/coldstart/pending_vertices/test.vertices.yaml -v C:\coldstart:/coldstart vesoft/nebula-importer:v2.6.0 --config /coldstart/pending_vertices/test.vertices.yaml
2022/03/18 04:46:49 --- START OF NEBULA IMPORTER ---
2022/03/18 04:46:49 File(/coldstart/pending_vertices/students/*.CSV) doesn't exist
2022/03/18 04:46:50 --- END OF NEBULA IMPORTER ---

YAML FILE

version: v2
description: example
removeTempFiles: false
clientSettings:
  retry: 3
  concurrency: 1 # number of graph clients
  channelBufferSize: 1
  space: StudentCentral
  connection:
    user: root
    password: nebula
    address: host.docker.internal:9669
  postStart:
    commands: |
      DROP SPACE IF EXISTS StudentCentral;    
      CREATE SPACE IF NOT EXISTS StudentCentral(partition_num=6, replica_factor=2, vid_type=FIXED_STRING(80));
      USE StudentCentral;
      CREATE TAG IF NOT EXISTS                      Student(sudentId string, hcs string, docInstance string);
maritalStatusId int, raceIds string);
    afterPeriod: 8s
logPath: /csv_data/err/test.log
files:
  - path: ./students/*.CSV
    batchSize: 10000
    inOrder: false
    type: csv
    csv:
      withHeader: false
      withLabel: false
      delimiter: ","
    schema:
      type: vertex
      vertex:
        vid:
          type: string
          index: 0
        tags:
          - name: Patient
            props:
              - name: sudentId
                type: string
              - name: hcs
                type: string
              - name: docInstance
                type: string

porscheme commented 2 years ago

@wey-gu can you please help me with this issue?

wey-gu commented 2 years ago

could you help with this?

run --rm -it --add-host host.docker.internal:host-gateway --network host -v C:\coldstart\pending_vertices\test.vertices.yaml:/coldstart/pending_vertices/test.vertices.yaml -v C:\coldstart:/coldstart vesoft/nebula-importer:v2.6.0 /bin/ls -l /coldstart/pending_vertices/students/*.CSV

Or, ensure those files are named .CSV rather than .csv ?

porscheme commented 2 years ago

I renamed .CSV to .csv; not worked
I tried this command and I can list the data files

C:\>docker run --entrypoint /bin/sh --rm -it --add-host host.docker.internal:host-gateway --network host -v c:\coldstart\pending_vertices\test.vertices.yaml:/coldstart/pending_vertices/test.vertices.yaml -v c:\coldstart:/coldstart vesoft/nebula-importer:v2.6.0
~ # ls /coldstart/pending_vertices/students/*.csv
/coldstart/pending_vertices/students/test.csv
~ #

wey-gu commented 2 years ago

Could you kindly help with this, to see if it's a bug only when the filename starts with *？

...
files:
-  - path: ./students/*.CSV
+  - path: ./students/t*.csv
    batchSize: 10000
    inOrder: false
    type: csv
    csv:
...

porscheme commented 2 years ago

fyi...I'm using v2.6.0 importer.

2022/03/18 07:11:13 --- START OF NEBULA IMPORTER --- 2022/03/18 07:11:13 File(/root/students/t*.csv) doesn't exist 2022/03/18 07:11:14 --- END OF NEBULA IMPORTER ---

porscheme commented 2 years ago

Upon further testing...

If there is only one file at the target location, wildcard in the file path doesn't work
If more than one file at the target location, wildcard in the file path does work fine

wey-gu commented 2 years ago

Thanks @porscheme , this should be a bug to be addressed, thanks! will keep this opened.

porscheme commented 2 years ago

How does this work in v3.0.1
Does v3.0.1 supported in open source K8S cluster deployment? I'm asking this Q because, helm charts are still referring to v2.6.1

wey-gu commented 2 years ago

This should be working on v3.0.1, too, with exactly the same importer config in v2.x
v3.x nebula-operator is not yet supported now, will be supported later :)

porscheme commented 2 years ago

Thanks @wey-gu for the reply.

If I understand correctly what you said, v3.0.1 and beyond open-source deployment will supported, can you confirm? We are correctly using v2.6.1 cluster; is there a way we can migrate to v3.0.1 right now? I will be on the Nebula community later today at 12:00 PM, can you join I have few questions?

wey-gu commented 2 years ago

I will join later :-), see you there!

wey-gu commented 2 years ago

v3.0.1 and beyond open-source deployment will supported, can you confirm?

wey: yes, I confirm it will be supported

We are correctly using v2.6.1 cluster; is there a way we can migrate to v3.0.1 right now?

wey: For now, in k8s, it cannot be migrated, while if it's bearemetal deployed, the upgrade/migration progress is in docs :)

wey-gu commented 2 years ago

Thanks @wey-gu for the reply.

If I understand correctly what you said, v3.0.1 and beyond open-source deployment will supported, can you confirm? We are correctly using v2.6.1 cluster; is there a way we can migrate to v3.0.1 right now? I will be on the Nebula community later today at 12:00 PM, can you join I have few questions?

For the k8s of 3.x, @MegaByte875 will share his draft PR when it's a little bit more mature before merging, if you are interested you could help test it soon.

Thanks!

veezhang commented 1 year ago

@porscheme Thanks! And it's already fixed in https://github.com/vesoft-inc/nebula-importer/pull/210 . And you can use the v2.6 branch .

QingZ11 commented 1 year ago

I have noticed that the issue you created hasn’t been updated for nearly a month, so I have to close it for now. If you have any new updates, you are welcome to reopen this issue anytime.

Thanks a lot for your contribution anyway 😊

vesoft-inc / nebula-importer

wildcard bug when matched file count is 1 #201