Unable to download the core context

NaIwo commented 2 years ago

I'm trying to run a docker instance of orion-ld based on: docker-hub. I do not have access to an external network for security reasons. Is it possible to run the image without access to the network? Could the error result from something else? Everything seems to work fine on the local computer with internet access. In addition to the standard context, we use hand-defined contexts in the project - but as I mentioned, everything has been tested locally on the machine and it works.

The logs during startup are as follows:

It seems like it can't get the relevant contextu definition metadata. Can we add them manually or maybe there is another solution/problem?

tassosgeo commented 2 years ago

Working today with @NaIwo to find a workaround for this, we discovered the following: If we deploy and startup orionld with mongodb on another machine that has access to public network, then context entries are created to the mongodb. Dumping this mongodb to the offline mongodb that is deployed in an air-gap infrastructure makes the orionld read the contexts bypassing its requests to the public context repo.

This is a workaround for now but it would be good if upon orion-ld deployment these context files can also be imported locally. Like mounting a volume containing them upon docker container creation.

kzangeli commented 2 years ago

When Orion-LD starts, if it cannot find the Core Context in mongo, it tries to download it. Without access to the core context, an NGSI-LD broker cannot function. Once it is downloaded and stored in mongo, no need to download anything anymore.

But, you want the broker to run without internet connection ... Perhaps you could mock the URL from where Orion-LD tries to download the core context?

@wistefan , you're implementing something similar about outgoing notifications ...

I implemented a "context cache on the local file system", for debugging purposes, but I think I deleted all that once I implemented the context persistence in mongodb ...

tassosgeo commented 2 years ago

@kzangeli we tried to mock the URL by changing the DNS entries of the Orion-ld container and serving the context files from a local web server. But it didn't work. Because the request is via https and it needs cert, a self-signed one was not sufficient as orionld request does not bypass an untrusted one currently. So with a trusted one it could work... but...

In this high security air-gap infrastructure the admins will not create this trusted certificate, meaning making a new one or an alias to an existing company's cert with a host like "uri.etsi.org" that already exists publicly.
Because this cert will be globally available. And it may be seen as the company has some sort of malicious-like behavior.

So mocking the URL is not possible in these high-secured infrastructures with a trusted cert.

kzangeli commented 2 years ago

ok, I'll invent something then. Perhaps make the URL to the core context a configuration parameter. It's just ... the core context is essential to the function of the broker. If I let users easy access to changing it ... disaster may be the result.

I understand your need though and I'll find a way (discussing it with my colleagues).

I'm in the process of creating the first ever real release of the broker and quite busy at the moment.

If you're in a hurry, I could help you to insert the core context in mongo "by hand", and your problem would be solved. Just let me know.

tassosgeo commented 2 years ago

@kzangeli Thanks for the support but we manage to make the mongo db dump with success! So everything is fine now.

I understand that it must not be easy for the users to change this configuration and the disaster that this will cause. I agree totally with you on this point.

But let's say an advanced configuration where some environmental variables are set first so that local context directory can be mounted, read from and auto-post context to mongo would be a possible solution.

I think something, that will be not the default behavior, should be done for this local context import, if Orion-ld is supposed to be deployed in industrial infrastructures.

I have in mind many cases where public network access is blocked in such infrastructures and it is quite common due security policies. They block access of sensors, robots, other devices and their data directly to public cloud and all the relevant services that are connected directly to them like orionld. They like lets say more to expose services that handle processed data and not raw data. Oriold can be used as data broker between public services but if you want Orionld to be used also to collect data directly from devices, offline deployment of it must be provided somehow as I believe this will be the case almost everytime in industrial production environments from my experience.

matteopantano commented 2 years ago

Hello @kzangeli @tassosgeo I am facing the same issue, how can I insert the core context by hand in Mongo?

kzangeli commented 2 years ago

Well, one way would be to temporarily let your node have internet access. Orion-LD tries to download the core context if it can't find it in the db. Once downloaded and stored in the database, you can close the internet access again (as long as you don't use any other contexts "out there").

It that is not possible, perhaps you can launch the broker on a different node, one with intertnet access, then mongo dump there, transport the dump file to your other node and do a mongo restore in there.

The contexts are stored in a tenant called "orionld".

matteopantano commented 2 years ago

Thanks @kzangeli, I will give it a shot like this and report how I did it so it is available to others for future reference ;)

kzangeli commented 2 years ago

Thank you Matteo. Do it well and I'll include it in the documentation of the broker :)

matteopantano commented 2 years ago

All right I was able to solve it. So this is how I did it. First I connected to the internet and run mongo-db togheter with orion-ld to be able to download all the context (I also let the cb download my own context by posting an entity with my context inside). Afterwards I used the command mongodump and I have saved the file locally.

My end goal was having orion-ld and mongo-db working without any internet connection, as long the docker image of mongo-db does not allow the persistence of the data I ended up solving the issue by creating my own docker image and installing mongo-db by hand with the following dockerfile:

# syntax=docker/dockerfile:1
FROM ubuntu:focal

RUN apt-get install wget && \
    wget --no-check-certificate -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | apt-key add && \
    echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse" | tee /etc/apt/sources.list.d/mongodb-4-4.list && \
    apt-get update && \ 
    apt-get install mongodb-org -y

COPY start.sh /home/start.sh
RUN chmod 755 /home/start.sh

# copy the local dump to a folder inside container
RUN mkdir -p  /home/mongo-orion-ld 
COPY <path-to-local-dump> /home/mongo-orion-ld 

RUN mkdir -p /data/db && \
    mkdir -p /data/log 

CMD /home/start.sh

where start.sh is the following:

#!/bin/bash

mongod --dbpath /data/db --fork --logpath /data/db/log --bind_ip_all && mongorestore --verbose /home/mongo-orion-ld 
bash

Through this I have a "mongo-db" image which has all the context inside and when I run the orion-ld no internet is required.

Ps: when running the fiware-orion-ld container remember to specify as -dbhost the name of the image you just build

FIWARE / context.Orion-LD

Unable to download the core context #952