New config file format for CAN devices

[PeterBowman]

All started with https://github.com/roboticslab-uc3m/yarp-devices/issues/74#issuecomment-530135206:

I'd fancy a .ini file structure that doesn't enforce properties for nodes which are never going to use them, e.g. maxVels in CuiAbsolute. Also, at some point in the future we might develop a motor drive node (analogous to TechnosoftIpos) which expects a different set of properties. Therefore, a plausible solution is to segregate those properties per CAN node type and place them in separate groups within the .ini file, e.g. [TechnosoftIpos_2], [CuiAbsolute_102] etc.. The CanBusControlboard wrapper should not need to parse and pass them along to wrapped subdevices, just take care of loading the .ini group on a per-subdevice initialization basis.

Beyond the YARP handling of .ini properties via C++ code, I'd like to rethink the config file format from scratch. See:

• https://www.yarp.it/yarp_config_files.html
• https://www.yarp.it/yarp_resource_finder_tutorials.html
• https://www.yarp.it/yarp-config.html
• https://www.yarp.it/yarp.html#yarp_resource
• http://wiki.icub.org/wiki/YARP_ResourceFinder
• https://github.com/robotology/robots-configuration
• https://github.com/roboticslab-uc3m/questions-and-answers/issues/61

[PeterBowman]

• experiments: find-group.zip

[PeterBowman]

We want to split mechanical device info from https://github.com/roboticslab-uc3m/teo-configuration-files and place it in a robot-agnostic repository much like https://github.com/robotology/robots-configuration. Perhaps "roboticslab-uc3m/configuration-files"?

The following scheme describes each mechanical component

Relevant info organized in directories:

• driver (e.g. iPOS)

  // ipos-sdfs.ini // driver
  name "ipos-sdfs" // dupe wrt filename?
  description "ipos-sdfs is a dfdsfsdfb that url: sdfsdfdsf"
  maxCurrent 10.0

• motor (e.g. Maxon EC motor)

  // maxxon586946.ini // motor
  name "maxxon586946-24v-1600rpm" // dupe wrt filename?
  description "maxxon586946-24v-1600rpm is a dfdsfsdfb that url: sdfsdfdsf"
  maxVelMech 10.0 // motor
  k 0.0706
  pulsesPerSample 1000

• transmission (e.g. Harmonic Drive)

  // hd-sdf.ini // transmission
  name "hd-sdf" // dupe wrt filename?
  description "hd-sdf is a dfdsfsdfb that url: sdfsdfdsf"
  tr 100.0

• relative encoder

  // cui-sdf.ini // relative encoder
  name "sdfsf" // dupe wrt filename?
  description "sdf is a dfdsfsdfb that url: sdfsdfdsf"
  encoderPulses 4096

• absolute encoder (...)

In <robot>-configuration-files, add one .ini per joint in which all remaining parameters are described, mainly motion variables and YARP device information. Everything strictly related to the robot or robot part. Also, the previous description files are referenced and thus aimed for translusion from within a YARP application.

// joint-sdg.ini
device TechnosoftIpos // controller? controllerDevice?
name "HSF-1" // dupe wrt filename?
description "sdg is a dfdsfsdfb that url: sdfsdfdsf"
canId 1
maxMech 90.0
minMech -90.0
max 90.0
min -90.0
maxVel 10.0 // joint, uses tr
refAcceleration 0.575437
refSpeed 5.0

driver ipos-sdfs
motor maxxon586946
transmission hd-sdf
relEncoder cui-sdf
absEncoder ...
// ..

[PeterBowman]

Several .ini files need to be loaded on different levels:

• canBusLauncher app
• CanBusControlboard device
• raw CAN subdevices (e.g. TechnosoftIpos)

TBD: either use .ini file transclusion (rather automatic and transparent to the device consumer), or traverse properties and perform file lookup with YARP's C++ APIs.

Edit: file and directory transclusion seems to work in a local fashion, only, i.e. YARP's resource finder mechanisms are not invoked whenever I put [include "file.ini"] or [include "dir"] (it does work with full absolute paths, though, but we surely don't want this). In conclusion, the full configuration file chain will need some serious C++ care (which I like more because of cleaner .ini files resulting from this). Note that context probably need to be switched on runtime via yarp::os::ResourceFinder::setDefaultContext, that is, it may not be doable from within each device as they are passed a yarp::os::Searchable instance, but from the omnibus generic canBusLauncher app instead.

Edit2: we still may take advantage from file transclusion as long as it is performed locally. For instance, a very long motor-maxon-asdf.ini could be split into motor-maxon-mechanical-data.ini, motor-maxon-thermal-data.ini, etc., and transcluded via [include] syntax in the main file. Potentially reusable by similar components. Note that this feature may not be supported in XML config file format, should we ever think about migrating from .ini to .xml.

[PeterBowman]

I'm willing to take advantage from the, currently unused by us, robots directory.

The resourcefinder spec covers a list of special data directories, among which we mostly exploit the <project>_CONTEXTS_INSTALL_DIR, e.g. ROBOTICSLAB-KINEMATICS-DYNAMICS_CONTEXTS_INSTALL_DIR. The yarp-config utility lists all available context directories and allows the user to copy them from the usual system paths (/usr/local/share) to local user paths (/home/user/.local/share). Then, yarp resource performs a lookup on said directories and tells us whether the file can be later located on runtime by the C++ application.

I'd like to place all component-related configuration files common to all robots into roboticslab-uc3m/configuration-files. Those will be installed in the usual context paths (share/<app>/contexts). Then, robot-specific stuff could be installed in a sibling robots directory (share/<app>/robots), e.g. <prefix>/share/roboticslab/robots/teo, where <prefix> is either a system or a user-local path. The roboticslab part originates from CMake code via yarp_configure_external_installation(roboticslab); currently, we pass it the lower-cased name of the repository. The robot directory, e.g. teo, should gather both models (teo-openrave-models) and launcher files (teo-configuration-files). In order to get resourcefinder to locate those robot-specific directories, the environment variable YARP_ROBOT_NAME should be set accordingly. Perhaps C++ code can be instructed to parse the desired robot via YARP option via yarp::os::ResourceFinder::getHomeRobotPath, assuming the robot configuration has been imported into the local user directory with yarp-config.

[jgvictores]

/usr/local/share/roboticslab-configuration-files
├── robots
│   └── teo
│       ├── openrave
│       │   ├── teo
│       │   ├── dextra
│       │   ├── lacqueyFetch
│       │   └── other
│       ├── gazebo
│       ├── bullet
│       ├── wrls (?)
│       ├── urdfs (?)
│       ├── joints
│       ├── nodes
│       ├── launch
│       └── kinematics
├── contexts
│   ├── drivers
│   ├── encoders
│   ├── motors
│   ├── sensors (!)
│   └── transmissions
└── applications
    ├── teo
    │   ├── teoBase
    │   └── follow-me
    └── asibot
        └── visual-servo

[jgvictores]

Option 1:

description "Axial left shoulder joint with iPOS drive (ASR)"
name "AxialLeftShoulder"
max 84.1
min -51.6
maxVel 50.0
refSpeed 10.0
refAcceleration 10.0
driver "ipos-3604-mx"
motor "maxon-411815"
encoder "cui-amt-203-v"
reverse true

gearbox "tr-hd-100"

[belt]
in "Optibelt ZRS 30-3M-6"
belt "OMEGA 255-3M 3"
out "Optibelt ZRS 44-3M-6"
dist 90 mm
N 1.47

Option 2:

belt-system "correa-tal"
gearbox "tr-hd-100"

--- teo/joints/belt-systems/belt-tal.ini
name "belt-tal"
in "Optibelt ZRS 30-3M-6"
belt "OMEGA 255-3M 3"
out "Optibelt ZRS 44-3M-6"
dist 90 mm
N 1.47

Option 3:

transmissions "belt-tal" "tr-hd-100"

---  roboticslab-conf-files/transmissions/belt-tal.ini
name "belt-tal"
in "Optibelt ZRS 30-3M-6"
belt "OMEGA 255-3M 3"
out "Optibelt ZRS 44-3M-6"
dist 90 mm
N 1.47

Option 4:

description "Axial left shoulder joint with iPOS drive (ASR)"
name "AxialLeftShoulder"
max 84.1
min -51.6
maxVel 50.0
refSpeed 10.0
refAcceleration 10.0
driver "ipos-3604-mx"
motor "maxon-411815"
encoder "cui-amt-203-v"
reverse true

transmissions "my-custom-1" "my-custom-2" "tr-hd-100"

[my-custom-1]
description "in Optibelt ZRS 30-3M-6, belt OMEGA 255-3M 3, out Optibelt ZRS 44-3M-6, dist 1.92"
N 1.47

[my-custom-2]
description "in Optibelt ZRS 30-3M-6, belt OMEGA 255-3M 3, out Optibelt ZRS 44-3M-6, dist 1.92"
N 1.47

• In any case, avoid unnecessary roboticslab-conf-files/optibelt-xxx.ini?

[PeterBowman]

Some research:

• .ini files are parsed in BottleImpl.cpp, there is no external library involved, just plain text exploded into individual characters and lots of loops and conditional clauses.
• .xml files are parsed with the tinyxml library, each application enforces its own format (e.g. yarpmanager supports a limited set of properly documented nodes and attributes).
• Everything in https://github.com/robotology/robots-configuration is meant to be read by the yarprobotinterface app (ref), which implements an XML parser via tinyxml.
• Parsed XML data is stuffed into "raw" C++ classes, structures and containers, then parsed again so that the iCub counterpart of our CanBusControlboard can reuse it (ref).
• All iCub files must follow a DTD specification, I can tell they had just two of them so far: version 1.0 and version 3.0.
• These files are entirely parsed and their information reused from C++ code, whereas we basically aim to store, for instance, component hardware specs which are mostly transparent to our apps.
• Each iCub fetches its own copy of yarprobotinterface launch data, even if there might be little or no difference wrt. other iCubs. In contrast, our https://github.com/roboticslab-uc3m/configuration-files are inherently reusable.
• The file tree seen by yarprobotinterface is recursively parsed via XML transclusion techniques (they introduced a convenient <include> tag, ref). We parse what we need via ResourceFinder's on-system file lookup (resembles the --from option).
• In fact, yarprobotinterface is the (much) more sophisticated counterpart of launchCanBus. I fancy its usage of the so-called "actions" (ref1, ref2): configure, calibrate, attach, abort, detach, park, custom.

[PeterBowman]

Had a long chat with @jgvictores.

Expanding on the all-in-one config file location, we tried to unify installation directories so that every roboticslab-uc3m project would copy its YARP config files into /usr/local/share/roboticslab. Most notably, we would end with highly populated plugins/ and contexts/ directories, but at least everything in the same place as opposed to having it scattered around (roboticslab-kinematics-dynamics/, roboticslab-vision/, etc.). Then, we realized that the mandatory yarp_configure_external_installation CMake command, when passed a roboticslab parameter under such assumption, would end up (re)creating and installing the same roboticslab.ini into /usr/local/share/yarp/config/path.d/. This crucial, shared file is deleted upon a sudo make uninstall command issued from anywhere within our org.

All in all, we backed off this idea and concluded that the current one-installation-dir-one-repo solution is gold. In that vein, the shiny new https://github.com/roboticslab-uc3m/configuration-files repository will die in flames, soon. Also, we were pondering about the fact that certain OpenRAVE models might be reused by other robots (e.g. Lacquey gripper), but given the lack of demand for this functionality, it is best to preserve the current status quo and get back to that only in case such need ever arises.

So, we'll keep things simple: configuration files next to stuff that uses them. Camera descriptors (e.g. asus-xtion.ini) in the vision repo, speech dictionaries in the speech repo, and hardware components as the ones introduced above (drivers, motors, encoders, transmissions...) hosted by the yarp-devices repo since the only thing that loads them, the launchCanBus app, is also meant to stay there.

Conclusions that affect things related specifically to this issue:

• launch .ini files will live in contexts/launchCanBus within <robot>-configuration-files; no collisions are expected (see https://github.com/roboticslab-uc3m/questions-and-answers/issues/61) since this app should only run in a single-robot environment
• leave kinematics-related stuff in its current location, we are currently hardcoding these anyway
• I'd like to give the robots/ directory a shot - unless C++ code gets overly complicated as a result

[jgvictores]

Two tiny comments, concerning non-existent cases, but as my opinion if they ever appear in the future:

• If an XXXXX repo's share/ directory or equivalent ever gets too big: I'm not against a meiosis resulting in a XXXX-resources repo (note that XXXX-resources would only have 1 consumer).
• If we ever find a specific case where the resources of one repository are consumed by several repositories (excluding 2 types of consumers that are not actually counted as consumers: {ROBOT_NAME}-configuration-files-sytle repos, and gitbook or jekyll-style documentation generators): we'll study the specific case.

PS: Re: "I'd like to give the robots/ directory a shot - unless C++ code gets overly complicated as a result" -> Sure, go for it! :smile:

[PeterBowman]

I'm flipping.

Facts:

• No other robot will reuse TEO's hardware (and if it does, replicating stuff is not that bad, see next points).
• Contrary to our initial thoughts, .ini file format is not as much user-friendly as an CSV is, for instance. This layout is meant for parsing by C++ code.
• Instead of shoving e.g. motor characteristics and lots of other non-relevant data (in the launchCanBus's POV) in them, I'd rather exploit our new datasheets-and-manuals repository.
• This actually brings us closer to robotology's robots-configuration repo in terms of usability. We can (and should) take advantage of file transclusion via [include] directives, as opposed to performing intensive filesystem lookup (I estimated almost 200 calls to ResourceFinder::findFileByName and Property::fromConfigFile alongside eachother).
• Then, teo-configuration-files should be the correct placement for TEO's launch files, not yarp-devices (previously, we considered configuration-files while seeking centralization).

[PeterBowman]

I'm glad we flipped the table. Following a robotology-like approach, each robot defines its own set of hardware&software description files (.ini format). Everything is parsed via transclusion beforehand, then conveniently reused by application code in dependent INIs (e.g. node INIs fetch information from shared motor INIs). Therefore, we avoid including the same file more than once and every (sub)device defines its own configuration format rules while taking advantage of a database-like collection of robot properties.

The root of this is a config.ini file installed into /share/robots/<robot>/. The launchCanBus app does not enforce any other requirement for robot-specific data, thus developers are free to choose the most suited format for their needs. Regarding TEO, I've adopted a directory layout that anyone can understand given the contents of the .ini file (ref):

// YARP devices
[include "buses"]
[include "calibrators"]
[include "nodes"]

// hardware-specific data
[include "hardware/drivers"]
[include "hardware/encoders"]
[include "hardware/motors"]
[include "hardware/transmissions"]

In order to avoid key overlapping, every .ini needs to define its properties within a group. ASIBOT devs may choose a completely different layout as long as config.ini loads everything (it does not need to use file/directory transclusion at all).

[PeterBowman]

Ready at https://github.com/roboticslab-uc3m/yarp-devices/pull/229/commits/f8d6c005c07de3d27c1aff3feb11cd827965bb95 and in conjunction with this branch at teo-configuration-files (currently at https://github.com/roboticslab-uc3m/teo-configuration-files/commit/2c49a57b4b1ad758cfbb57de2583daea5b250510).

[PeterBowman]

Follow-up: https://github.com/roboticslab-uc3m/yarp-devices/commit/5e3ed52447a0fb4151225cd21553e0b7bb0b462d. It turns out a yarp::os::Property invalidates all references (especially those created via addGroup) whenever this same instance is used to populate another Property via fromString.

[PeterBowman]

Oh, by the way. That's how you unmarshall a blobbed (blobbified?) pointer:

https://github.com/roboticslab-uc3m/yarp-devices/blob/5e3ed52447a0fb4151225cd21553e0b7bb0b462d/libraries/YarpPlugins/CanBusControlboard/DeviceDriverImpl.cpp#L26

This actually avoids a deep copy of the original object every time makeBlob is called:

https://github.com/roboticslab-uc3m/yarp-devices/blob/5e3ed52447a0fb4151225cd21553e0b7bb0b462d/programs/launchCanBus/LaunchCanBus.cpp#L81

[PeterBowman]

The YARP_ROBOT_NAME=teo line has been added to /etc/environment in both manipulation and locomotion TEO PCs so that this variable is set on system startup.

[jgvictores]

The YARP_ROBOT_NAME=teo line has been added to /etc/environment in both manipulation and locomotion TEO PCs so that this variable is set on system startup.

Not the best place, but added at https://github.com/roboticslab-uc3m/teo-developer-manual/commit/e2b2e64781a82d4c10667a08151f0f0df8b22ddc

[PeterBowman]

Commit https://github.com/roboticslab-uc3m/yarp-devices/commit/8082d0c4713ad7fc70a8d3d99f05ce415c57bd55 allowed to parse common TechnosoftIpos/CuiAbsolute options gathered in common-ipos.ini and common-cui.ini, respectively, thus reducing the overall file size.