Performance issue

[Neeraj-Nec]

I am using this library for expanding and compaction the NGSILD data but getting issue of performance for more than 30 parallel request . my request is getting failed by error message "not able to decode in valid json"

[kazarena]

@Neeraj-Nec,

The only place in the library where we decode JSON directly is when we load remote documents, JSON-LD contexts, primarily. Before I can look into the issue, can you please answer the questions below?

1. Are you caching JSON-LD contexts between different calls to the library? DocumentLoader provides several caching strategies.

2. Can you please share a sample JSON-LD document you are expanding and compacting? Just the top level structure, with "@context" field, would suffice.

3. What is the exact error message?

Regards, Stan

[Neeraj-Nec]

Than you for you response This is my data . I am using link context

curl -iX POST \ 'http://180.179.214.202:8070/ngsi-ld/v1/entityOperations/upsert' \ -H 'Content-Type: application/json' \ -H 'fiware-service: openiot' \ -H 'fiware-servicepath: /' \ -H 'Accept: application/ld+json' \ -H 'Link: https://fiware.github.io/data-models/context.jsonld; rel="https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld"; type="application/ld+json"' \ -d ' [ { "id": "urn:ngsi-ld:Vehicle:VehicleA01", "type": "Vehicle", "brandName": { "type": "Property", "value": "Mercedes" }, "isParked": { "type": "Relationship", "object": "urn:ngsi-ld:OffStreetParking:Downtown1", "providedBy": { "type": "Relationship", "object": "urn:ngsi-ld:Person:Bob" } }, "speed": { "type": "Property", "value": 30 }, "location": { "type": "GeoProperty", "value": { "type": "Point", "coordinates": [-8.5, 41.2] } } } ]'

[Neeraj-Nec]

please find the error :

err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld

[kazarena]

@Neeraj-Nec,

I didn't get the answer to the first question, but I believe the problem is that your JSON-LD contexts aren't cached. As a result, every invocation of the JSON-LD library functions, like expansion or compaction, makes a request to the context URL (as seen in the Link in your example). At some rate, https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld fails to load with some error message instead of valid JSON. I would recommend using either RFC7324CachingDocumentLoader or CachingDocumentLoader. It will both substantially increase performance and address the failure you are seeing.

[Neeraj-Nec]

May be you are correct . can you please provide some document link how to use CachingDocumentLoader.

[Neeraj-Nec]

I was use this link to expand https://github.com/piprate/json-gold/blob/master/examples/expand.go

[Neeraj-Nec]

The code that I have written func (tb *ThinBroker) ExpandData(v interface{}) ([]interface{}, error) { proc := ld.NewJsonLdProcessor() options := ld.NewJsonLdOptions("") //LD processor expands the data and returns []interface{} expanded, err := proc.Expand(v, options) return expanded, err } is :

[Neeraj-Nec]

May be catching document loder load the document locally and use it further . if it is then please help how to use it.

[kazarena]

@Neeraj-Nec, see the example below how to use NewRFC7324CachingDocumentLoader. You just need to pass it in the options.

package main

import (
    "log"

    "github.com/piprate/json-gold/ld"
)

func main() {
    // this instance of the document loader should be shared
    // across all library requests that benefit from caching.
    dl := ld.NewRFC7324CachingDocumentLoader(nil)

    proc := ld.NewJsonLdProcessor()

    opts := ld.NewJsonLdOptions("")
    opts.ProcessingMode = ld.JsonLd_1_1
    opts.DocumentLoader = dl

    doc := map[string]interface{}{
        "@context":  "http://schema.org/",
        "@type":     "Person",
        "name":      "Jane Doe",
        "jobTitle":  "Professor",
        "telephone": "(425) 123-4567",
        "url":       "http://www.janedoe.com",
    }

    expanded, err := proc.Expand(doc, opts)
    if err != nil {
        log.Println("Error when expanding JSON-LD document:", err)
        return
    }

    ld.PrintDocument("Result", expanded)
}

[Neeraj-Nec]

Thanks !! i think by the same way we can compact the expanded data by using compact function?

[kazarena]

Thanks !! i think by the same way we can compact the expanded data by using compact function?

Yes.

[Neeraj-Nec]

Still getting some request failed ! with the same above error . but no of failed request is decreased . i am using jmeter to test using 50 parallel request. and rampup period in second is 1

[Neeraj-Nec]

err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld err2 loading remote context failed: Dereferencing a URL did not result in a valid JSON-LD context: https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld

[kazarena]

@Neeraj-Nec the errors below suggest you aren't using the same document loader across the requests.

[Neeraj-Nec]

document you mean request! every request have different id and type

[kazarena]

I mean the instance of the DocumentLoader. Another possibility that your test harness ramps up so quickly that the DocumentLoader doesn't have a chance to cache the context (it's non-blocking, as far as I remember). In this case, you may want to pre-cache the context.

[Neeraj-Nec]

I am getting what is document loader.

i have replace the code according suggestion

func (tb *ThinBroker) ExpandData(v interface{}) ([]interface{}, error) { proc := ld.NewJsonLdProcessor() dl := ld.NewRFC7324CachingDocumentLoader(nil) opts := ld.NewJsonLdOptions("") opts.ProcessingMode = ld.JsonLd_1_1 opts.DocumentLoader = dl expanded, err := proc.Expand(v, opts) //options := ld.NewJsonLdOptions("") //LD processor expands the data and returns []interface{} //expanded, err := proc.Expand(v, options) return expanded, err }

[kazarena]

Yes, that's exactly the problem. You are creating a new instance of the DocumentLoader with every call. You need to create it outside of the function, and then pass it in the Options.

[Neeraj-Nec]

OK so only one instance is sufficient for every call . can you please suggest when to Crete new instance of document loader

[Neeraj-Nec]

I think if @context is different from previous context the new instance is required

[kazarena]

OK so only one instance is sufficient for every call . can you please suggest when to Crete new instance of document loader

@Neeraj-Nec, it depends on the architecture of your application. I would recommend confirming first that the issue is indeed in too frequent requests to the underlying context URL. If confirmed, the provided DocumentLoaders should be sufficient to address it. If they don't suit your architecture, it's easy to write your own document loader.

[Neeraj-Nec]

I have only one context that is in the request. On the other hand I will check it with my team

[Neeraj-Nec]

But for the time being we are using only one context. And for this I think one instance is sufficient

[Neeraj-Nec]

Thanks Kazarena! The suggested implementation is working for me for the provided @context . Now I will discuss the architecture with my team for other @context.

[Neeraj-Nec]

Still getting same above issue

i have change the code

func (tb *ThinBroker) ExpandData(v interface{}) ([]interface{}, error) {
    //var dl *ld.RFC7324CachingDocumentLoader
    dl := Expand_once()
    proc := ld.NewJsonLdProcessor()
    opts := ld.NewJsonLdOptions("")
    opts.ProcessingMode = ld.JsonLd_1_1
    opts.DocumentLoader = dl
    expanded, err := proc.Expand(v, opts)
    //LD processor expands the data and returns []interface{}
    //expanded, err := proc.Expand(v, options)
    return expanded, err
}

func Expand_once() *ld.RFC7324CachingDocumentLoader {
    if ldE == nil {
        ExpandOnce.Do(
            func() {
                ldE = ld.NewRFC7324CachingDocumentLoader(nil)
                fmt.Println("created object", ldE)
            })
    } else {
        fmt.Println("The loader object is already created")
    }
    return ldE
}

[Neeraj-Nec]

Do you have any document how to test other feature of documentLoader.go

[kazarena]

@Neeraj-Nec is there a chance you are hitting the Expand function (and consequently, remote context loading) simultaneously from different clients/goroutines? If so, the cold cache will try to load the remote context multiple times, since there is no locking internally. In this case, I would recommend pre-populating the loader's cache by calling LoadDocument() for all well-known contexts that you use in your application before exposing it to high load.

[Neeraj-Nec]

ya you are right . i have many clients more than 500 simultaneously. The library is giving above mention error. do yo have any document how to implement your recommendation.

[Neeraj-Nec]

https://github.com/smartfog/fogflow/blob/8bf5c833fe3c26f7e21b654b319dc0e5f4684860/broker/thinBroker.go#L2744 this is link of my project where i am using it .

[kazarena]

There is no need for a document. You may just pre-load the required context when you construct the loader:

 _, err := dl.LoadDocument("https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld")

Again, it depends on the architecture of your application as where and how to do it.

[Neeraj-Nec]

if i already have preloaded document then is there any of option . and how i can expand my data by preloaded document i am getting the document by the code _, err := dl.LoadDocument("https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld") how to utilize use this document to expand my json data

[Neeraj-Nec]

How to expand our message based on this preloaded document

[kazarena]

@Neeraj-Nec,

Conceptually, you may do it this way:

func Expand_once() *ld.RFC7324CachingDocumentLoader {
    if ldE == nil {
        ExpandOnce.Do(
            func() {
                ldE = ld.NewRFC7324CachingDocumentLoader(nil)
                fmt.Println("created object", ldE)

                _, err := ldE.LoadDocument("https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld")
                // check the error
            })
    } else {
        fmt.Println("The loader object is already created")
    }
    return ldE
}

[Neeraj-Nec]

if i am loading the document the document by this _, err := ldE.LoadDocument("https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld") . then i think threre is no need to provide @context in message witch is going to be expand.

[Neeraj-Nec]

proc := ld.NewJsonLdProcessor() options := ld.NewJsonLdOptions("")

// expanding remote document

expanded, err := proc.Expand("http://json-ld.org/test-suite/tests/expand-0002-in.jsonld", options) if err != nil { log.Println("Error when expanding JSON-LD document:", err) return }

// expanding in-memory document

doc := map[string]interface{}{ "@context": "http://schema.org/", "@type": "Person", "name": "Jane Doe", "jobTitle": "Professor", "telephone": "(425) 123-4567", "url": "http://www.janedoe.com", }

expanded, err = proc.Expand(doc, options)

In the above example doc := map[string]interface{}{ "@context": "http://schema.org/", "@type": "Person", "name": "Jane Doe", "jobTitle": "Professor", "telephone": "(425) 123-4567", "url": "http://www.janedoe.com", } contain @context

[Neeraj-Nec]

I think you are right expansion is totally depend on architecture of my project . but can you please help how to prepare dl object if i have array of @context in one message.

[kazarena]

@Neeraj-Nec,

If you choose to use the strategy above, any external context can and should be preloaded using LoadDocument(...).

Alternatively, if your contexts are static, you may want to use CachingDocumentLoader as shown here.

[Neeraj-Nec]

No i do not have static @context . i think i need to write the document Loader for my project

[kazarena]

By static contexts, I meant the contexts that change rarely.

If you feel you need a custom DocumentLoader to satisfy your requirements, it's certainly a way.

[Neeraj-Nec]

@kazarena define @context totally depend on the user .

[Neeraj-Nec]

Do you have any function/method that deserialize the expanded data.