Is it possible to Data Record All Sim Variable without writing them in input.py

[Fjolnirr]

Hello,

I have developed a mongodb adapter in the sim_services called DRMongo.cpp . It basically works just like DRAscii however it records to mongodb collection instead of csv. My purpose is to record all sim variables by the given dataRecord cycle. (My question down at the end)

Currently, i have created a script to fill up the input.py where my input.py looks like below:

exec(open("Modified_data/realtime.py").read())

print("This is trick-->")
drg = trick.DRMongo("Fjorn")

drg.set_mongoDbUri("mongodb://localhost:27017")

drg.add_variable("dyn.cannon.pos[0]")
drg.add_variable("dyn.cannon.pos[1]")
drg.add_variable("dyn.cannon.vel[0]")
drg.add_variable("dyn.cannon.vel[1]")

drg.freq = trick.DR_Always
trick.add_data_record_group( drg , trick.DR_Buffer)

trick.exec_set_terminate_time(1*20)

Simply i am adding variables line by line. This is the resulting document in the mongodb:

{
  "_id": {
    "$oid": "65cb31acc988a2eaa90e2413"
  },
  "sys.exec.out.time": "                 0.1",
  "dyn.cannon.pos[0]": "  0.4330127018922194",
  "dyn.cannon.pos[1]": "           0.2495095",
  "dyn.cannon.vel[0]": "   43.30127018922194",
  "dyn.cannon.vel[1]": "             24.9019"
}

It is obviously is not recording the parameter in the document format it records as string parameter(this is a problem for mongo when trying to read those parameters somewhere else). Well I can use some regex and string manipulations to make those string params like this:

"sys": {
    "exec": {
      "out": {
        "time": "                   0.1"
      }
    }
  },

However, using input.py with lines of parameters to read from dataRecord buffer is sth else.

What I want to know is if trick has a function or location that i can grab all the simulation variables and its corresponding values at once each cycle?

[hchen99]

trick-sie & trick-dre can show you all variables as you see on trick-tv (left hand side). Example on using trick-dre to create data recording file: https://nasa.github.io/trick/tutorial/ATutRecordingData.html

Btw, is DBMongo very necessary as csv files can be easily imported to MondoDB?

[Fjolnirr]

Well I have used trick-dre too but still i need to select all the variables that i want to record from the list. I need something like DataRecordGroup::add_variable but like DataRecordGroup::add_all_variables or something similar. If you have any suggestion like this is not a dataRecord place but ... place to look for i would be happy to look at. Since the simObjects has the struct/object that holds the data maybe you could point me where to read that data without requesting from dataRecord.

To answer your question: I have a backend and frontend project that is connected to the trick. I am handling variable server operations(start, freeze shutdown etc.) over my backend and isolating the frontend. I have seen the react frontend of yours too but it uses directly the trick socket where i have other operations for my project that requires backend. Frontend is basically sum of sim control panel and trick tv apps. I am gathering S_sie.resource and transform it to json format so it is easier for my frontend to parse. I was first using the variable server to collect data with my backend over socket and send it to my frontend. But i find it more practical to read real time data from mongodb. (I have a structure that whenever an insert happens in db, mongo is triggered and my service receives the data and also since the service triggered it sends the incoming data over websocket where my frontend receives it.)

It is doable with socket of course however i want to isolate the simulator from my project. Also recording to a csv file over hours/days with cycle of 0.1 sec might break it so i believe database is safer. And uploading the csv with a script almost same as doing it manually is something that i want to avoid.

[hchen99]

No single function gives you all variable without you setting things up first. Since you already parsing S_sie.resource, you can set things up programmatically easily. Thought you just wanted the recorded data which is to be put into MongoDB. Thus the question. Thanks for sharing what you're trying to do. Sounded like you tried different ways and want to stick with your existing setting for getting data whenever db inserts instead of getting data directly from trick variable server. Intuitively, I'd think if you could get data directly why getting data, saving it to database and then retrieving data from the database? Not sure if you tried trick.var_server_create_tcp_socket(...) to create another socket for your data so you can get data directly?

[Fjolnirr]

To be honest, all the time i thought and questioned just like you but recording all at once to a db kind of my requirement. Since you have commented I could persuade the people i am working with by referencing your comments. My first goal was to implement, DR and TV panels just like in trick but with a variable service(nodejs) and frontend TV&DR pages(reactjs). Before sim;

• select the params to record to db,

and

On sim;

• select the params to watch real time over trick variable server (no recording or record the selected params to as a new collection).

I mean my mongodb implementation is not an empty work right now. Actually with this setup i mentioned It might be a quite a solution and if you consider an alternative to csv recording by just providing the port, trick will record directly with a proper json format to database, I can open a pull request when i finish all my tests. My only problem was to record all data in the simulation but I mean if it was efficient i am sure you guys would implement it by now. (when trying to record all data i have record ~65.000 parameters to db at 0.1 cycles. As far as I know in DR documentation it says ~100.000 parameters is acceptable.)

I have not tried or seen this one trick.var_server_create_tcp_socket(...) I definitely will look at it. Thanks for your precious comments.

[hchen99]

Sounded a fun project. The thought might not work for your requirement but could be fun to play with it. Recording all variables definitely needs to be careful in terms of performance, file/db/disk size, and etc. However, should be unlimited number of parameter references. By default, the max records to hold in memory before writing for a recording group is 100,000 which can be set to a different number.

[amishscientist]

You could drop checkpoints at a fixed rate. In most cases that would be overkill for what you want. Fixed rate checkpoints is definitely trying to solve the problem with a bazooka. A lot of times I can write scripts to auto generate code by parsing the sie file or an ASCII checkpoint file.