romnn / mongoimport

CLI and go library for importing data from CSV, JSON or XML files into MongoDB.
MIT License
3 stars 2 forks source link

Issue on closing closed channel #18

Open norbertogomez opened 3 years ago

norbertogomez commented 3 years ago

Using your solution as part of a repository in a web server, the first API call does the import in a successful way. On the second call, looks like the library is trying to close an already closed channel.

The error I am getting:

panic: close of closed channel
marketing-feed-manager | 
marketing-feed-manager | goroutine 585 [running]:
marketing-feed-manager | github.com/gosuri/uiprogress.(*Progress).Listen(0xc000041680)
marketing-feed-manager |    /go/pkg/mod/github.com/gosuri/uiprogress@v0.0.1/progress.go:117 +0x108
marketing-feed-manager | created by github.com/gosuri/uiprogress.(*Progress).Start
marketing-feed-manager |    /go/pkg/mod/github.com/gosuri/uiprogress@v0.0.1/progress.go:134 +0x3f

The way I use the mongoimport library:

    conn := mongoimport.MongoConnection{
        AuthDatabaseName: m.AuthDatabase,
        DatabaseName:     m.Database,
        Host:             m.Host,
        Password:         m.Password,
        Port:             m.Port,
        User:             m.User,
    }

    datasources := []*mongoimport.Datasource{
        {
            Description:  "All offers",
            FileProvider: &files.List{Files: []string{filepath}},
            Options: mongoimport.Options{
                Collection: m.Collection + market.Code,
            },
        },
    }

    csvLoader := loaders.DefaultCSVLoader()
    csvLoader.Excel = false
    importer := mongoimport.Import{
        Sources:    datasources,
        Connection: &conn,
        Options: mongoimport.Options{
            EmptyCollection:    opt.SetFlag(true),
            IndividualProgress: opt.SetFlag(false),
            Loader:             loaders.Loader{SpecificLoader: csvLoader},
            FailOnErrors:       opt.SetFlag(false),
            PostLoad: func(loaded map[string]interface{}) ([]interface{}, error) {
                return []interface{}{loaded}, nil
            },
        },
    }

    result, err := importer.Start()

After a debugging session, I think the issue could be in here

jaumebecks commented 3 years ago

I think this is related to https://github.com/gosuri/uiprogress/issues/40 @romnn Specifically when initializing a UI progress bar in https://github.com/romnn/mongoimport/blob/4b28142843284f4c870f40c196e55bea22c9c773/import.go#L79, you might use uip := uiprogress.New() instead