Error getting package: pattern must be a valid regex

[fils]

Using the code at: https://github.com/fils/THROUGHPUTDataPackages/tree/master/dpex

Which is just a simple app to read and open a firctionless data set to work with the raw read.

At line 14 of file: https://github.com/fils/THROUGHPUTDataPackages/blob/master/dpex/outfitter/outfitter.go

I get

fils@xps:~/src/go/src/oceanleadership.org/frictionless/THROUGHPUTDataPackages/pkgpress$ go run main.go
2018/03/18 10:00:04 About to listen on 9990. Go to http://127.0.0.1:9990/
2018/03/18 10:00:14 Error getting package: pattern must be a valid regex

I'm really perplexed here. These seems to be an error come from outside the datapackage-go package but am not sure from where or why?

go version go1.10 linux/amd64 Ubuntu 16.04 x86_64 GNU/Linux

I feel like I missing something simple... but I can't see it.

[danielfireman]

Hi @fils , good afternoon.

Taking a look at the dpex/data/data/carpLakeCoreStratigraphy.csv, I noticed that Depth below lake surface column is not composed only by numbers (looks like data bellow line 22 should be another column), which will cause an error.

Another source of problems could be the use of the type text for Notes and Core Segments to Examine. Text is not a valid type as per tableschema specification. Please use string

Finally, the tableschema project was not tested against Go 1.10, which I've already done.

I totally agree that the error message is quite bad. Could you please try those suggested changes? We could change this issue to have more descriptive error messages if your tests pass after the fixes in the data schema.

[fils]

@danielfireman ugh.. how embarrassing. I sorta forgot I'd never bothered to make these in to proper packages. I'll try and fix these in the morning and test.

However, I did pull down the simple-geojson package from the example packages repository and tested with it. I still seem to get the same error. I'll test things more in the morning and try and update this issue with some proper examples and links.

[fils]

@danielfireman So I modified the code and put in

pkg, err := datapackage.Load("https://raw.githubusercontent.com/frictionlessdata/example-data-packages/master/simple-geojson/datapackage.json")
    if err != nil {
        log.Printf("Error getting package: %v\n", err)
        return "Error getting package"
    }

in https://github.com/fils/THROUGHPUTDataPackages/blob/master/dpex/outfitter/outfitter.go at line 15

I get the same results

2018/03/19 09:07:56 About to listen on 9990. Go to http://127.0.0.1:9990/
2018/03/19 09:08:03 Error getting package: pattern must be a valid regex

any guidance on what I am doing wrong here?

[fils]

@danielfireman Just wanted to give some more details. I'm getting the same issue on my Mac with Go 1.9.1

Also, when I remove the error trapping I get the following

Fils:dpex dfils$ go run main.go
2018/03/21 03:17:04 About to listen on 9990. Go to http://127.0.0.1:9990/
2018/03/21 03:17:09 http: panic serving 127.0.0.1:57481: runtime error: invalid memory address or nil pointer dereference
goroutine 7 [running]:
net/http.(*conn).serve.func1(0xc420085720)
    /usr/local/go/src/net/http/server.go:1697 +0xd0
panic(0x13ff740, 0x16d1510)
    /usr/local/go/src/runtime/panic.go:491 +0x283
github.com/frictionlessdata/datapackage-go/datapackage.(*Package).GetResource(0x0, 0x1476a6d, 0x7, 0x0)
    /Users/dfils/src/go/src/github.com/frictionlessdata/datapackage-go/datapackage/package.go:47 +0x26
oceanleadership.org/frictionless/THROUGHPUTDataPackages/dpex/outfitter.SchemaFromPackage(0x0, 0x0)
    /Users/dfils/src/go/src/oceanleadership.org/frictionless/THROUGHPUTDataPackages/dpex/outfitter/outfitter.go:21 +0x90
oceanleadership.org/frictionless/THROUGHPUTDataPackages/dpex/lphandler.PresentPackage(0x16a3a20, 0xc4200fe1c0, 0xc42016e600)
    /Users/dfils/src/go/src/oceanleadership.org/frictionless/THROUGHPUTDataPackages/dpex/lphandler/lphandler.go:13 +0x34
net/http.HandlerFunc.ServeHTTP(0x14a0598, 0x16a3a20, 0xc4200fe1c0, 0xc42016e600)
    /usr/local/go/src/net/http/server.go:1918 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc4200805a0, 0x16a3a20, 0xc4200fe1c0, 0xc42016e600)
    /Users/dfils/src/go/src/github.com/gorilla/mux/mux.go:109 +0xdc
net/http.(*ServeMux).ServeHTTP(0x16dc6a0, 0x16a3a20, 0xc4200fe1c0, 0xc42016e100)
    /usr/local/go/src/net/http/server.go:2254 +0x130
net/http.serverHandler.ServeHTTP(0xc4200fa410, 0x16a3a20, 0xc4200fe1c0, 0xc42016e100)
    /usr/local/go/src/net/http/server.go:2619 +0xb4
net/http.(*conn).serve(0xc420085720, 0x16a4220, 0xc42005bc40)
    /usr/local/go/src/net/http/server.go:1801 +0x71d
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2720 +0x288

I'm trying to track down where this "pattern must be a valid regex" is coming from. I have never used delve before.. but I might give it a shot.

Thanks! Doug

[fils]

I tried a stand alone test.. to ensure I wasn't messing up other elements... I still get the error

fils@xps:~/src/go/src/opencoredata.org/ocdGarden/frictionlessdata/rawread$ go run main.go
Raw Read Test
Load error
pattern must be a valid regex
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x714c26]

goroutine 1 [running]:
github.com/frictionlessdata/datapackage-go/datapackage.(*Package).GetResource(0x0, 0x7e3f03, 0x7, 0x1e)
        /home/fils/src/go/src/github.com/frictionlessdata/datapackage-go/datapackage/package.go:47 +0x26
main.main()
        /home/fils/src/go/src/opencoredata.org/ocdGarden/frictionlessdata/rawread/main.go:20 +0xde
exit status 2

with the self contained code....

package main

import (
        "fmt"
        "github.com/frictionlessdata/datapackage-go/datapackage"
        "io/ioutil"
)

func main() {

        fmt.Println("Raw Read Test")

        //pkg, err := datapackage.Load("data/datapackage.json")
        pkg, err := datapackage.Load("https://raw.githubusercontent.com/frictionlessdata/example-data-packages/master/simple-geojson/datapackage.json")
        if err != nil {
                fmt.Println("Load error")
                fmt.Println(err)
        }

        so := pkg.GetResource("example")
        rc, err := so.RawRead()
        if err != nil {
                fmt.Println("rawread error")
                fmt.Println(err)
        }

        defer rc.Close()
        contents, err := ioutil.ReadAll(rc)
        if err != nil {
                fmt.Println("ioutil error")
                fmt.Println(err)
        }

        fmt.Println(contents)

}

[danielfireman]

Hi Doug, sorry for the delay to answer. After a ton of debugging (including change the library we use to parse JSON schemas), I believe the problem is twofold:

• https://frictionlessdata.io/schemas/data-resource.json (the canonical source of schemas) has been updated and this change included an invalid pattern (that explains your error message)
• our schema registry mechanism had a bug wasn't picking up the local cache instead of the remote one (that was fixed).

Could you please update your datapackage-go and let me know how it goes?

[fils]

@danielfireman I just check out both my codes using rawread and they are working as expected!

Thank you SO much!

From my POV the issue can be closed!

Take care!