Simulator for ev3dev or ev3dev-python

[DJuego]

Hi!

I have ten hours of experience with ev3dev-python and I am pretty pleased. In fact I am considering seriously to use python-ev3dev in a educational environment.

The only weakness that I find is lacking of a simulator for learning/experimenting/working without robot.

Of course, to develop an accurate emulator is a HUMONGOUS project (!!!). However a toy simulator , similar to Open Roberta simulator would be a fantastic addition: A 2D top-down view of a two drive wheels mobile robot offers potential for learning and it is an inmediate visual feedback

Any related projects (robotic simulators in python): here, here and here

Any thought, suggestion, hint, idea or link in this direction? 👍

DJuego

P.S V-Rep seems promising. Would it be adapted?

[WasabiFan]

This is a very interesting concept. Firstly, I think that something like this would need to be implemented at the language binding/user program level, as both emulating the OS and drivers in situ would be a lot of work to do efficiently without a bunch of installation work. However, I think it would be relatively easy to modify the existing Python library so that file I/O could be replaced with proper calls to a simulated environment (preferably something that could be re-used by others as well). However, such an environment would need a "mock" implementation of the ev3dev motor and sensor drivers; this presents the challenge of keeping them in sync with changes to the real ones, because nuances can make a large difference. Optimally, you'd want a way to compile the existing drivers standalone for userspace so that you could both feed them your own input and intercept writes to PWM for motors. @dlech am I correct in assuming that there is no means of doing this that wouldn't cause difficulties in core driver maintenance? Do you have any ideas in this area?

In the end, I think that all the wrapper logic around the motor drivers would be easy, assuming any physics would be limited to 2D and an simple robot model. It's just a matter of getting the realistic logic working that translates ev3dev driver inputs to motor power outputs.

[ensonic]

@DJuego did you try the OpenRoberta system (https://lab.open-roberta.org). This has a built-inb simulator for the ev3/nxt. It is a bit higher level (we're using this in classrooms with kids staring 3rd grade).

[DJuego]

Thank you for your comments, @WasabiFan, Yes, yes. That is the idea. In the same vein of what i was thinking. Perhaps it is more easy to use the remote API of V-Rep. See this example.

@ensonic: Yes, yes. I know OpenRoberta System. :-) It is a powerful tool (intuitive, free, online but it is possible to work offline,...). In fact, I request to translate that simulator concept to ev3dev-python.

P.S:@ensonic, Do you use ev3dev with OpenRoberta? Does it work well? Or is it better to use the operating system for Open Roberta? What is your experience?

[ensonic]

@DJuego are you sometimes in IRC or the gitter chat? Don't want to hijack this bug report for discussion. Anyway, as quick pointers:

• works offline: run the openroberta server on a raspberry pi 3 (I am doing this for events)
• using ev3dev: you need a custom firmware for the ev3 anyway, if you use lejos, openroberta generates java, if you use ev3dev then openroberta generates python. I wrote the ev3dev connector for openroberta (https://github.com/OpenRoberta/robertalab-ev3dev) and I've been using it for quite a few events.

[dlech]

@dlech am I correct in assuming that there is no means of doing this that wouldn't cause difficulties in core driver maintenance? Do you have any ideas in this area?

I would not recommend using Linux drivers as part of the simulation stack. It would limit you to running simulations on Linux only and add unnecessary complexity.

However, I think it would be relatively easy to modify the existing Python library so that file I/O could be replaced with proper calls to a simulated environment

I think this is the "right way" to go about doing something like this.

[WasabiFan]

The main issue that I see here is the ev3dev simulation logic itself: This type of virtual environment would need to have an implementation of the motor drivers available to translate driver input to duty cycle. Do you have any suggestions there? Or is it simple enough that an alternate implementation could be written and parity maintained without issue?

[dlech]

This type of virtual environment would need to have an implementation of the motor drivers available to translate driver input to duty cycle

Not necessarily. If you are telling the motor to run at a fixed speed, then just have the simulation run the motors at that speed. If you are making a simple OpenRoberta-style simulator, there is no need to model the actual physics of the motor. For example, if speed_sp=400 and the simulation has a wheel diameter of 20 pixels, then the robot should be moving across the screen at 400 tacho counts per second / 360 tacho counts per rotation * (3.14 * 20) pixels per rotation = 69.8 pixels / second. Duty cycle does not come in to this equation at all.

If you want a simulator that is fancier than that, then you are beyond my expertise. :smiley:

[WasabiFan]

The problem is that you need to expose something resembling the interface of the real drivers. That means that commands, polarity, position, input constraints, PID dynamics (for positional control), ramping, state and stop actions would all need to be operational as they are in the real environment for consistent results. You could, of course, decide that it would only support the speed-control mode, immediate breaking, "running" state, etc., but at that point the simulation would probably not actually simulate real circumstances and would fail when you tried to use more than just the bare essentials of functionality.

I'll have to think over how this could work. There isn't an obvious answer that I can come up with but maybe I'll think of something.

[dwalton76]

Of the simulators linked above do we know if any them support a plugin architecture where we could swap out them emulating say EVG-3 for a block of Python code?

[dwalton76]

I started playing around with making a very basic simulator. The init goal is just to have a single large motor appear as connected, will worry about simulating functionality later.

The problem is that we need to write to /sys/ which you cannot do from user space, even as root. What if we could trick the kernel into thinking a motor is connected...that would get us past the inability to write to /sys/.

@dlech when a motor is connected how does the kernel detect that it is connected?

[dlech]

You could write a dummy kernel driver to simulate the sysfs structure, but I think it would be much easier to just replace the functions that wrap calls to sysfs with something else. If this simulator is going to run on non-Linux systems, then that is also your only choice. FWIW, you can find the EV3 output port device connection manager here.

[WasabiFan]

In my mind, I think the ideal structure would be to intercept calls one level up: make an abstraction layer between the core library code and file I/O, then have an implementation of that layer which calls a separate Python module which operates the simulator. That avoids the file I/O and works cross-platform. Preferably the simulator code would expose an interface which is also usable by other languages.

[ensonic]

Can we mount an overlayfs to sysfs to shadow the real files? Or we have a fusefs replicating the relevant sysfs structure and the language bindings try two locations (first the fusefs one, then the sysfs one). We could even use some API to set the prefix and thus select between real and emulated.

[WasabiFan]

What I'd like to know is: Why would we want to emulate the files in any fs? If our goal is to make a simulator that runs on the desktop, isn't making it dependent on a Linux kernel counterproductive? I think we should be designing this thing to be entirely platform-independent, and cut out all the file I/O in the simulator.

[dwalton76]

I was going to take the ev3dev bin and run it in a virtual box VM, I figure the closer we are to the real thing the better and virtual box will run on windows, linux and mac. My thought was the user launches the emulator then launches their robot code without making any changes to the robot code.

I think dlech makes a good point though that it might just be easier to have the python library look somewhere other than /sys/ if we want to run in emulator mode. So the user would

• start the emulator which creates the needed filesystem structure somewhere other than /sys/
• start the robot but pass it some flag on the command line to tell it to look at the emulated directory instead of /sys/

Perhaps then we wouldn't need virtual box.

[dwalton76]

@WasabiFan if the filesystem is emulated then we shouldn't need to change anything in ev3dev-lang-python other than telling it to look at the emulated directory instead of /sys/. That seems much easier than adding a code path to ev3dev-lang-python that allows it to function without file I/O. Maybe I am missing something though, what are your thoughts on how to do the simulator without file I/O?

[ensonic]

@dwalton76 an alternative would be to do this in the platform probing and have a new emulated platform

[rhempel]

Am I missing something? Don't we just need to change

''' DEVICE_ROOT_PATH = '/sys/class' '''

In 'core.py'

[WasabiFan]

Am I missing something? Don't we just need to change

No, you're not missing something; if we want to have the library write to files and then the simulator read from them, that'll work. However, I am advocating for entirely bypassing the file I/O and intercepting calls between our external library interfaces and the I/O calls; that way we aren't using the hard drive for data transfer between modules, which seems nonsensical.

[dwalton76]

I'll be honest I am not connecting the dots yet on what clear advantage we get by avoiding file I/O. Sure we could come up with something that avoids file I/O but why are we concerned about file I/O? My gut says we could pull off an emulator pretty quickly if we just point DEVICE_ROOT_PATH at a directory that is being manipulated by the emulator and in theory we shouldn't have to change anything else in ev3dev-lang-python. But if we want to do an emulator that avoids file I/O I don't see how we could do that without significant changes in ev3dev-lang-python.

[WasabiFan]

I'll be honest I am not connecting the dots yet on what clear advantage we get by avoiding file I/O.

Disk I/O is not an interface to be used for live communication. Hard drives and their related software are designed for long-term storage, and as such they don't work well when you want to talk to other usermode software: it doesn't immediately invoke the other app, it puts needless pressure on system resources, and it wears down the physical media. Maybe this is a difference stemming from the fact that I've learned with Windows while you have with *Nix; it's important to keep in mind, however, that sysfs and memory-mapped files are not the same as real files on disk.

But if we want to do an emulator that avoids file I/O I don't see how we could do that without significant changes in ev3dev-lang-python.

The only I/O calls that motors and sensors invoke comes from the Device class. All we'd need is an alternative implementation of three methods. In reality, implementing this simulator while bypassing file I/O is likely easier than writing one that has to handle high-frequency reads and file watching.

[dwalton76]

I think part of our misunderstanding here is assumption that file I/O implies disk I/O. which as you pointed out, is not always the case in linux given filesystems like tmpfs which are using ram and not a hard drive. So if we went the "emulate the filesystem" route we would certainly want to avoid writing to the SD card all the time and would use tmpfs or some equivalent. /var/run/ is one place we could do this, it uses tmpfs.

In terms of using the filesystem (not the disk) for communication, user space apps do this all the time with the kernel, the fact that we would have two user space apps communicating this way isn't a bad thing IMO. Heck even if we had the emulator communicate with the ev3dev-python code via a unix domain socket that is still just a file in the file system.

[dwalton76]

The only I/O calls that motors and sensors invoke comes from the Device class. All we'd need is an alternative implementation of three methods. In reality, implementing this simulator while bypassing file I/O is likely easier than writing one that has to handle high-frequency reads and file watching

But if we do this then the way you troubleshoot a robot changes a good bit if it is being emulated. I often do a cat of a file in /sys/class/tacho-motor/ to get the position of a motor or I echo a number into one of those files to write some value.

If the emulator and the app are not communicating via the filesystem in the same way the kernel and the app would communicate, then that is pretty fundamental change even if it is only three methods in the Device class that would need to be changed.

[dwalton76]

Something else that just came to mind, if we simulate the filesystem then it should be trivial for the other ev3dev programming language repos (Go, C, C++, etc) to also take advantage of the emulator.

[WasabiFan]

I think I'm starting to see where our thinking diverged.

In my mind, an ev3dev simulator as described in this issue is software that allows you to run your ev3dev code on a development PC (whether that happens to be a desktop, laptop or dead badger, running any Python-supported OS) without the need for an EV3 or ev3dev installation available. There's no "SD card" in that scenario, and file I/O must inherently be assumed to be disk I/O (unless you fancy writing file system drivers for the aforementioned dead badger).

What have others been imagining?

[dwalton76]

I guess I was picturing a virtual box image of ev3dev (plus the emulator), that way everything is as close as we can get it to the real thing. If they need to "apt-get install python-opencv" in the VM to make their robot work then they know they'll need to do the same thing on their ev3.

[Dave-Baum]

A virtual box is going to get as close as possible to running on real hardware. There are certain kinds of problems that will only show up that way. But there is also a lot of value in a less faithful simulation that runs natively on your development computer. Easier setup, using your IDE's debugger will be a breeze, etc. From my perspective, if real hardware is available then the primary advantage of a simulator is faster iteration, better debugging, and setup of test environments. I suspect all of these are going to be simpler and easier when running native.

With native simulation there are still two basic options:

1) Manipulate stuff in the filesystem namespace so that ev3dev runs unmodified, except for perhaps a new root for where to look for files. This has a good architectural boundary and allows the simulator to be used with any language. I think the biggest risk is coming up with a portable and efficient mechanism to watch a hundred files for writes.

2) Introduce an abstraction layer near the bottom of python ev3dev/core.py that allows the simulation to patch in its own attribute objects. They can still reside in a filesystem like namespace, but they are going to behave more like device drivers (albeit written in python). Super easy to make this portable (and pure python), but the boundary isn't quite as clean. I wrote a very simple proof-of-concept by monkey patching ev3dev. I think the approach is sound and would be cleaner if we introduced a real abstraction layer instead of patching. I think this also paves the way for fast unit tests of ev3dev and/or user code.

[loopspace]

I'd like to chime in on this thread from an education point of view.

What would be really useful to have is a python package that could be imported into a script on a PC and used to do a simple sanity check on a program before it is transferred to the device running ev3dev. This would enable the students to check for standard errors (indentation, missing colons, 'true' instead of 'True') without the overhead of transferring to the device.

It wouldn't have to be all that accurate, and wouldn't have to provide any sort of graphical/textual demonstration of what was going on. These would be nice, but I'd go with the maxim that the perfect can be the enemy of the good, and say that something that simply allowed program validation would be a huge benefit here.

Education environments tend to be locked down, so a simple module would be the best solution here.

[dlech]

http://stackoverflow.com/questions/4284313/how-can-i-check-the-syntax-of-python-script-without-executing-it

[Dave-Baum]

Compiling is a good way to catch indentation and syntax errors. I don't think it will catch true vs. True, or most other cases of typos in variable/function/method names.

[ghost]

What about re-born RoboSim?

• https://github.com/InfoSphereAC/RoboSim

[dwalton76]

@WasabiFan @ddemidov @dlech we've had this open for 14+ months and none of us has gotten to it, any objections to closing this one? We can re-open if someone has the urge to work on it.

[dlech]

I don't see any harm in leaving it open - just tag it as "wishlist" or something like that so people know it is not likely to be implemented.

[alanwilter]

Sorry if it sounds I am hijacking this thread, but my idea for simulator right now would be something like this:

• I have a backup, on my Mac, of the SD Card flashed and configured with eve3dev;
• I would like to use this backup file to load in a, say, Virtual Box EV3 brick simulator, and update, install etc. using a likely much faster environment with steady internet connection;
• then flash back the updated backup on my SD card and put it on my brick.

The point for me here is that EV3 Brick hardware is very slow for, say, upgrading ev3dev kernel and debian packages. It's time consuming and battery draining.

Do we have anything around that could do this?

[dlech]

@alanwilter: It will take you much longer than running things on the EV3, but it is possible. It works like this:

1. on a linux machine (or virtual machine) mount the EV3DEV_ROOTFS partition somewhere
2. mount the EV3DEV_BOOT partition to boot/flash of the rootfs that you just mounted
3. copy qemu-arm-static to usr/bin in the rootfs
4. poke a few more files to prevent services from trying to run when you install packages
5. poke some more files to make flash-kernel work correctly
6. chroot into the rootfs and upgrade your packages
7. exit the chroot and undo all of the poking and copying of files that you did
8. unmount the partitions
9. put the SD card back in the EV3 and hope you didn't break anything

Probably better to invest in some rechargeable batteries.

[Zincfox]

Hey, I just thought I'd let you guys know that I'm giving this a shot. Unfortunately it's been quite some time since I've used python but what's the worst that could happen? I already made some progress and will make my repo public once I feel like I have something presentable. I do not plan to add anything fancy like a position tracker or the ability to use duty-cycle (at least currently). The GUI will just consist of the Brick, the screen/display + buttons and various slots for the sensors (probably interactive) and motors. My current approach is to have a "host" process and a "client" process which can communicate over a multiprocessing-Pipe. The host will start the client and hold the current robot-state and also do the rendering. The fs-methods in init.py (ev3dev2) will be overwritten by the client process before the main code starts by "catching" the import of the init.py and replacing the methods of the delivered file with the simulator bound version. This has two advantages: 1) The "real" and the "simulated" robot code are separated - no checking for simulators on the actual brick and no trying to access the filesystem (aside from the code itself of course) on the simulator. 2) Backwards-compatibility. Because the modification is "in place", the original source code doesn't need to be modified - I hope that it's possible to support ev3dev and ev3dev2 with this setup (the different names allow for loading different "patches" - if the methods fluctuated to much in the past it should in theory also be possible to use version-files you seem to have in place).

I also have the idea of using a "progressive" simulation model instead of an iterative one. An example would be the Motor-component: Instead of incrementing the rotation by the current speed each tick, I simulate the rotation as a (currently linear) mathematical function. Setting values recalculates the parameters for these functions (the slope for rotational speed) and sets the current time. Then every time those values are read the current value is calculated by using the (now different) time. This should make the robot behave more realistically as we don't risk loosing "simulation ticks". Currently this is only implemented for motors and obviously isn't appropriate for every device - the buttons should obviously just stay atomic.

What do you guys think? I abstracted the simulator-simulatee gap into an "connector" class which should make it possible to switch to fs- or port-based communication later on to support other languages if this works out.

[dwalton76]

Sounds great to me 👍 Looking forward to trying it out

[cho934]

Sorry if it sounds I am hijacking this thread, but my idea for simulator right now would be something like this:

• I have a backup, on my Mac, of the SD Card flashed and configured with eve3dev;
• I would like to use this backup file to load in a, say, Virtual Box EV3 brick simulator, and update, install etc. using a likely much faster environment with steady internet connection;
• then flash back the updated backup on my SD card and put it on my brick.

The point for me here is that EV3 Brick hardware is very slow for, say, upgrading ev3dev kernel and debian packages. It's time consuming and battery draining.

Do we have anything around that could do this?

It would be very interesting to update the ev3dev by taking the sd card inside a virtual machine, because it's very very long !

[ghost]

https://www.aposteriori.com.sg/Ev3devSim/index.html

I did not make this.

[ndward]

Related to the previous comment... cort@aposteriori.com.sg has created a simulator for ev3dev python... https://www.aposteriori.com.sg/Ev3devSim/index.html

In his own words from a message he sent me (I haven't yet tried the simulator myself, though it looks exciting):

It simulates a 2-wheeled differential drive robot commonly used in competitions. I made it mainly to help my students practice programming python on the ev3 even when they don't have access to the actual robot. Figured it'll be useful to others as well. The simulator runs entirely in the browser, so it can scale to any number of simultaneous users without significant load on the server (...just need to download the maps, javascript files, etc).

BTW, Cort also made EV3fast, – a Python module for faster ev3 interface, which he introduced in 2018 here: https://github.com/ev3dev/ev3dev-lang-python/issues/451

[WasabiFan]

Yeah, that simulator is quite cool. I'll have to look into it to see exactly how it works sometime. Beware it's probably based on their own binding API which won't match exactly with ev3dev-lang-python.

[QuirkyCort]

Hi All,

I made the simulator for much of the same reasons that @DJuego needed it. It also helps with preparing for competitions like FLL, WRO, and Robocup Rescue. Didn't know about this thread until @ndward told me about it, and there are some great ideas here for implementing the simulator.

The block programming interface in OpenRoberta is still something that I would use for younger kids who are not ready for Python. Unfortunately, the simulator doesn't run ev3dev-lang-python code and the simulated robot is not configurable, limiting its utility.

@WasabiFan, you're right that the simulator uses it's own bindings. I try to get it as close as possible to ev3dev-lang-python, and consider any differences to be a bug on my end. Some differences may be unavoidable, as the simulator runs Python code using the Skulpt interpreter inside the browser and not cPython.

The simulator is not meant to fully replace testing with a real robot. While it is possible for a simulator to model more characteristics of the robot (eg. PID, motor responses, inertial, backlash), there are also many factors which are impractical to simulate either because it'll require the user to input difficult to obtain information (eg. differences in each individual motors/sensors, friction between wheels and ground), or the behaviour is unpredictable (eg. tire skiding). Instead of trying to make the simulator more accurate, I think that it's important for students to understand that real robots don't behave with 100% accuracy and consistency, and be able to write programs that can handle these inconsistencies (eg. using feedback loops to align the robot).

It's a weekend project (...albeit a long one; we just had a 4 days weekend in Singapore this month), and I'm sure it is still lacking in many areas. If anyone has a request for new features or bugfixes, do let me know. Any help would also be appreciated (...especially on the color sensor; I can't find any documentation on how the ev3 sensor map between RGB and color, and the current implementation is simply my best guess).

[QuirkyCort]

Uploaded the simulator to github.

https://github.com/QuirkyCort/ev3dev-sim

[dwalton76]

That looks really cool :)

[Zincfox]

The simulator I mentioned last year is now working (not polished), but I am not sure if I can make it open-source…

A bit of background: I am a student at the Technical University Dresden where I tutor a mandatory “RoboLab” course (students build and program ev3s which have to explore a labyrinth). This past semester I was able to get credits for my work on the simulator which allowed me to put a lot of work into it – this however also meant that a lot of code specific to the RoboLab-course has made it into that repository. I will have to discuss with my supervisor whether I can make this code public as it might make tasks which the students have to implement easier (specifically odometry), the next meeting is scheduled for the end of this month. In theory it would be possible to extract the parts needed for the RoboLab into a separate repository, but the tight integration of odometry to determine the position in the simulation would mean that many useful features would have to be moved as well (color-sensor feedback depending on position in map, etc.).

I implemented pretty much everything I mentioned earlier, including the interactive ui, ramping (as a second degree polynomial of motor-position), position-tracking (and as mentioned, the sensor feedback for color, gyro, distance and touch (only with fixed, circular obstacles for the last two)). I however also made a lot of physics-simplifications like @QuirkyCort mentioned. The file-connector is not implemented and as such the simulator can only be used to run python-code, but the abstraction is there. The simulator is not intended to be used by our students to develop their robot, but has already been used by the tutors to save grades when the physical demonstration failed because of hardware-problems. As such it is intended to run a wide variety of code (and robot-layouts) in a perfect-world-scenario.

In its current form the simulator is not useful for competitions involving line-following because the map is generated from our custom labyrinth-format and as such does not allow the use of image-files (and its data-provider is intended to be used by line-followers using only one color-sensor, unlike the model in aposteriori). It would be possible to register additional data-providers to tackle these problems though.

I used placeholder-images in one of the screenshots because I’m not sure how Lego would feel about me using their images here. Also the LEDs are still missing.

[QuirkyCort]

Hi All,

It's been nearly a year since I released Ev3DevSim, and many embarrassing bugfixes later, I finally got around to working on a new simulator https://gears.aposteriori.com.sg and github https://github.com/QuirkyCort/gears

Unlike the previous simulator, this one...

• Runs in 3D
• Performs physics simulation (...you can crash into walls and run off ramps now...)
• Support Blocks, ev3dev2-python, PyBricks
• Supports any number, combinations, and position of sensors (eg. 5 ultrasonic and 4 color)
• Have configurable actuators (eg. arm)
• Have a magnet (...doesn't exist for the EV3, but makes for more fun in the simulator)

Like the previous 2D simulator, this is free and opensource, and will always remain free and opensource.

It is still very much a work in progress and many things still don't work. Figure I'll put it out here to solicit comments, help, and suggestions

[WasabiFan]

Very cool, nice work!

[glipR]

Hello,

In light of everyone being stuck at home, we also made our own simulator - this is made with pygame/pymunk and can be used from the command line rather than through the web. It's also more focused on support for the Robocup Soccer rules than individual Rescue course. You can check it out (still in a prerelease stage) on github or pypi!

As some added features:

• time.time and time.sleep are mocked to give the simulator time - works for pausing and timescale.
• We mocked ev3devpython's attribute_file methods to get the simulation working, and use grpc for communications between robots and simulator - so if someone wants to add some other language support it should also handle this fine! (We are also working on transpiling Mindstorms code into ev3dev python at the moment)

Any feedback on this would be much appreciated!