search

OpenVisualCloud / Smart-City-Sample

The smart city reference pipeline shows how to integrate various media building blocks, with analytics powered by the OpenVINO™ Toolkit, for traffic or stadium sensing, analytics and management tasks.
BSD 3-Clause "New" or "Revised" License
189 stars 81 forks source link

Provisioning sensor script fails #754

Closed sudhirkshirsagar closed 3 years ago

sudhirkshirsagar commented 3 years ago

It is not possible to run provision.py in its directory because it references other py files in other directories.

xwu2git commented 3 years ago

Note that the smtc_db_init image derives from the smtc_common image.

sudhirkshirsagar commented 3 years ago

The image dependency does not resolve script dependencies. In the same image, the scripts need to have references to the correct directories for the Python script imports to work. How does one modify the provisioning after deployment when the images are built?

On Fri, Mar 19, 2021 at 9:05 PM xwu2 @.***> wrote:

Note that the smtc_db_init image derives from the smtc_common image.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OpenVisualCloud/Smart-City-Sample/issues/754#issuecomment-803241056, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD52ZGF73N7M7PPMJ4ZOPTLTEQNIRANCNFSM4ZP37P3A .

xwu2git commented 3 years ago

What script dependency is missing?

BTW, provision.py is just an example of initial provisioning. In a real product, a field provisioning tool must be written to dynamically insert provisioning information into the DB. This field tool is out of scope of the sample. Thus provision.py is there to write something to the DB so that the sample can function.

sudhirkshirsagar commented 3 years ago

from db_ingest import DBIngest is one example of the dependencies. That script is in common. Although a full reprovisioning may be out of the scope, the sample appears to have a very specific one-shot architecture. Also the sensors from sensor-info.json don't seem to get provisioned. It is not clear how to get them provisioned in the initial provisioning.

On Sat, Mar 20, 2021 at 2:26 PM xwu2 @.***> wrote:

What script dependency is missing?

BTW, provision.py is just an example of initial provisioning. In a real product, a field provisioning tool must be written to dynamically insert provisioning information into the DB. This field tool is out of scope of the sample. Thus provision.py is there to write something to the DB so that the sample can function.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OpenVisualCloud/Smart-City-Sample/issues/754#issuecomment-803465603, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD52ZGFM6XMCIXB3N4LB2G3TEUHHLANCNFSM4ZP37P3A .

xwu2git commented 3 years ago

Yes, db_ingest import DBIngest, which is in the smtc_common. This is why the db_init image uses smtc_common as the base image. The final built image smtc_db_init is self-sufficient with all the dependencies. In other images where DBIngest is needed, they either use smtc_common as the base image or copy the *.py scripts over. What was the issue exactly?

As for provisioning, the camera info from sensor-info.json is written to the provisions index. Then at runtime, when a new sensor is discovered, the IP address or URL is compared to the provisions index to find a match. Then the discover sensor is written to the sensors index. See sensor/discovery for details.

sudhirkshirsagar commented 3 years ago

Python scripts can't reference scripts in other directories. They need to have exact reference to those directories. Copying the required scripts is an option but not the best option. If following the tutorial steps does not work a single Xeon server where all the requirements have been installed then I suspect that there are dependencies that are not being addressed in the published version, or perhaps a single server install has never been tested in spite of what the tutorial says. The master builds but the v21.3 build fails. So that is another data point that the architecture is not fully tested for a single server docker swarm install.

On Sun, Mar 21, 2021 at 9:36 AM xwu2 @.***> wrote:

Yes, db_ingest import DBIngest, which is in the smtc_common. This is why the db_init image uses smtc_common as the base image. The final built image smtc_db_init is self-sufficient with all the dependencies. In other images where DBIngest is needed, they either use smtc_common as the base image or copy the *.py scripts over. What was the issue exactly?

As for provisioning, the camera info from sensor-info.json is written to the provisions index. Then at runtime, when a new sensor is discovered, the IP address or URL is compared to the provisions index to find a match. Then the discover sensor is written to the sensors index. See sensor/discovery for details.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OpenVisualCloud/Smart-City-Sample/issues/754#issuecomment-803617840, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD52ZGGUMVTI3PA5K5SQERTTEYOCLANCNFSM4ZP37P3A .

xwu2git commented 3 years ago

Don't fully understand what you say. All python scripts are under /home. No issue with execution. Single node docker swarm is tested. The v21.3 branch does not build simply because the required openvisualcloud software stacks have not yet released and uploaded to dockerhub.

sudhirkshirsagar commented 3 years ago

What image are you referring to when you run provision.py in a built image? Obviously the testing did not follow the tutorial method for a single node/server swarm on Ubuntu 18.04. Everything builds, all services run but you get errors in the services!! A branch should be able to be tested and run, otherwise there should be comments in the readme. Maybe there are. Rebuilding the analytics did not do anything.

On Mon, Mar 22, 2021 at 9:20 AM xwu2 @.***> wrote:

Don't fully understand what you say. All python scripts are under /home. No issue with execution. Single node docker swarm is tested. The v21.3 branch does not build simply because the required openvisualcloud software stacks have not yet released and uploaded to dockerhub.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OpenVisualCloud/Smart-City-Sample/issues/754#issuecomment-804200747, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD52ZGCS5HADUUODCS6222TTE5U35ANCNFSM4ZP37P3A .

xwu2git commented 3 years ago

Let's use the smtc_db_init image as an example. You can run into the image to talk a look:

docker run --rm -it smtc_db_init ls -l /home

You can see that all the python scripts and their dependencies are under /home.

sudhirkshirsagar commented 3 years ago

The smtc_db_init image shows the scripts in the home but clearly sensors from the json are not being provisioned. The env also shows lat long for the office variable as shown in the following, and the provisioning script has no comments so impossible to follow. Also it has variables like office1. bash-4.2$ printenv HOSTNAME=498cc4b2b5c4 TERM=xterm REPLICAS=0,0 NO_PROXY= PYTHONIOENCODING=UTF-8 DBHOST=http://db:9200 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin OFFICE=45.539626,-122.929569 GWHOST=http://traffic_office1_gateway:8080 PWD=/home SHLVL=1 HOME=/home SCENARIO=traffic no_proxy= DBCHOST=http://cloud_gateway:8080/cloud/api/db _=/usr/bin/printenv OLDPWD=/ bash-4.2$

dbhost=env["DBHOST"] office=list(map(float,env["OFFICE"].split(","))) gwhost=env.get("GWHOST",None) scenario=env["SCENARIO"]

def Provision(officestr): print("Provisioning...", flush=True)

# populate db with simulated offices and provisionings
with open("/run/secrets/sensor-info.json",encoding='utf-8') as fd:
    data=json.load(fd)
    dbp=DBIngest(index="provisions", office=office, host=dbhost)
    for office1 in data:
        if scenario != office1["scenario"]: continue
        location1=office1["location"]
        if location1["lat"]!=office[0] or location1["lon"]!=office[1]:

continue

        sensors=office1.pop("sensors")
        for s in sensors:
            s["office"]=location1
            if "ip" in s: # convert IP to CIDR
                if s["ip"].find("/")<0:
                    s["ip"]=s["ip"]+"/32"
                s["ip_text"]=s["ip"]  # dup for terms aggs
        dbp.ingest_bulk(sensors, refresh="wait_for")

        office1.pop("scenario")

On Mon, Mar 22, 2021 at 10:48 AM xwu2 @.***> wrote:

Let's use the smtc_db_init image as an example. You can run into the image to talk a look:

docker run --rm -it smtc_db_init ls -l /home

You can see that all the python scripts and their dependencies are under /home.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OpenVisualCloud/Smart-City-Sample/issues/754#issuecomment-804266903, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD52ZGHJ2KXLGVXF27G4VVDTE57INANCNFSM4ZP37P3A .

xwu2git commented 3 years ago

Please check deployment/docker-swarm/secret.m4 and deployment/docker-swarm/analytics.m4. The sensor-info.json is mounted to the image vs docker secrets.

sudhirkshirsagar commented 3 years ago

Here are the files. These are all created by the standard build as defined in the tutorial.

secrets: self_key: file: ../certificate/self.key self_crt: file: ../certificate/self.crt sensor_info: file: ../../maintenance/db-init/sensor-info.json

include(platform.m4)

ifelse(defn(SCENARIO_NAME'),traffic', defn(OFFICE_NAME')_analytics_traffic: image: PLATFORM_IMAGE(defn(REGISTRY_PREFIX')smtc_analyticsobject'defn(PLATFORM_SUFFIX')_'defn(`FRAMEWORK'):latest) volumes:

ifelse(defn(SCENARIO_NAME'),stadium', defn(OFFICE_NAME')_analytics_entrance: image: PLATFORM_IMAGE(defn(REGISTRY_PREFIX')smtc_analyticsentrance'defn(PLATFORM_SUFFIX')_'defn(`FRAMEWORK'):latest) volumes:

On Mon, Mar 22, 2021 at 1:34 PM xwu2 @.***> wrote:

Please check deployment/docker-swarm/secret.m4 and deployment/docker-swarm/analytics.m4. The sensor-info.json is mounted to the image vs docker secrets.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OpenVisualCloud/Smart-City-Sample/issues/754#issuecomment-804375645, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD52ZGCEBKYSAC2D73UHOB3TE6SXDANCNFSM4ZP37P3A .