search

cyberbotics / webots

Webots Robot Simulator
https://cyberbotics.com
Apache License 2.0
3.13k stars 1.66k forks source link

Quality Assurance R2020a #1164

Closed DavidMansolino closed 4 years ago

DavidMansolino commented 4 years ago

Tests:

Installation

Test Description
Installation Install Webots from the provided package, following the installation instruction from the Webots user guide.
Check that when pressing the simulation reset while the simulation is running you have the same behavior at the second iteration.Check this for the 10-15 first worlds of the guided tour.
Creation of a small project From scratch and using only the Webots GUI (wizards (including VisualStudio on windows), scene tree, new, and so on), create a world
VRML import/export Check that the VRML import/export menu is working.
Make a movie Make a movie using the menu: Files > Make Movie ...
Test at least a normal screen resolution and the full screen resolution (1920x1080 Full HD).
Delete objects in the Scene Tree 1. Open a complex world
2. Select the last object in the Scene Tree
3. Hit the [Delete] button until the Scene Tree is empty (besides the 3 top nodes)
4. Revert and repeat the operation starting from the 4th node.
The test succeeds if Webots does not crash.
Copy a robot in the Scene Tree 1. Open a world that contains a robot
2. Select the robot
3. Hit the [Copy] button
4. Hit the [Paste After] button
5. At this point there should be 2 robots close to one another
6. Separate the 2 robots using the mouse (if needed)
7. Save the .wbt
8. Push [Revert]
9. Push [Run]

The test succeeds if both robots have the same appearance and behavior, etc.
Java controllers 1. Open the world: projects/languages/java/worlds/examples.wbt
2. Hit [Run]

The test succeeds if all 3 robots move, the camera images change and there is no error message.

3. Modify Help message (Driver.java at line 88)
4. Hit the [Save] button
5. Accept the relocation of the project proposed by Webots, in that order hit: [Yes], [OK], [Open], [OK].
6. Hit the [Build] button
7. When Webots proposes to 'Revert', hit [Yes].
8. Hit [Run]

The compilation test succeeds if the help message printed in the console contains the last modifications.
Python controllers 1. Open the world: projects/languages/python/worlds/examples.wbt
2. Hit [Run]

The test succeeds if all 3 robots move, the camera images change and there is no error message.

All the Python versions officially supported by the OS should be tested.
Robot window The generic robot window opens itself correctly, even if the simulation is stopped or at step 0 (by double clicking on the robot, or by selecting the corresponding item in the Robot menu)

Note: The e-puck, the Thymio II and the DARwIn-OP have a custom robot window.
Devices worlds Run all the examples in the 'projects/samples/devices/worlds' directory and check that they all work as expected.
Support of UTF-8 characters Test that a user having UTF-8 characters in his account name can:
- launch Webots
- launch controllers
- Compile controllers
Visual Studio wizard Test that the VisualStudio wizard works.
Matlab controllers 1. Open the world: projects/languages/matlab/worlds/e-puck_matlab.wbt
2. Hit [Run]

The test succeeds if:
a) the e-puck robot moves
b) the camera and display viewports show moving images
c) a Matlab Window with 4 subwindows appears and all subwindows show moving graphics
Mouse interactions Using the mouse and the keyboard, ensure that one can move the camera (translate, rotate, zoom), the objects (xz translation, lift), and apply forces/torques.

A right click in the 3D scene opens the context menu.

A double click on a robot opens the robot window.
e-puck - remote control Try to remote control an e-puck2 robot over Bluetooth
Binary compilation Test that controllers / physics plugins / controller plugins (remote control / robot windows) / libraries (qt_utils) can be cleaned and compiled smoothly. On Windows, please make sure to test by opening Webots from the icon.
DARwIn-OP to test:
-> the simulations do the expected thing
-> controllers can be modified
-> you can remote-control and remote-compile the real robot
-> same with Robotis OP2
Make HTML5 animation Can create an animation (File / Make HTML5 animation) and play it into a web browser.
Streaming Server 1. start with the streaming server: webots --stream

2. open the web page located in resources/web/streaming_viewer/index.html and connect to Webots (using the default parameters).

3. the 3D view of Webots should display properly on the web page. It should be possible to change the viewpoint and select objects.
Extern controllers Open a simulation with a robot. Set the controller of the robot to and launch the controller from the command line. The test succeeds if the robot behaves the same as with the normal controller.
Offline documentation Check that the offline documentation is working from 'Help / offline documentation / ...' and from the context menu.
SUMO interface Open the 'projects/vehicle/highway' world:

1.Start the simulation and check that SUMO is correctly started in GUI mode.
2. Check that the version of SUMO (top of the SUMO GUI window) is the one specified in the Changelog.
3. Check in Webots that other vehicles are appearing and moving.
4. Check that the main vehicle is visible in SUMO as a green triangle (you may need to zoom).
OSM importer and SUMO exporter 1. Install the python dependencies as explained in https://www.cyberbotics.com/doc/automobile/openstreetmap-importer#dependencies

2. Convert the OSM file:

mkdir -p $HOME/my_project/resources/osm_files
mkdir -p $HOME/my_project/resources/OSM_importer
cp $WEBOTS_HOME/projects/vehicles/importer_demo.sh $HOME/my_project/importer_demo.sh
cp $WEBOTS_HOME/resources/osm_importer/config.ini $HOME/my_project/resources/OSM_importer/config.ini
cp $WEBOTS_HOME/projects/vehicles/resources/osm_files/village.osm $HOME/my_project/resources/osm_files/village.osm
cd $HOME/my_project
chmod u+x ./importer_demo.sh
./importer_demo.sh village

3. Open the generated world file and check the result is similar to the one distributed in Webots (project/vehicles/village).
4. In pause mode, add the SumoInterface node and set it's 'useNetconvert' field to FALSE.
5. Run the simulation and check that you can see vehicles appearing and correctly aligned on the roads.
Backward compatibility Open few worlds of the previous Webots version, and check they can be open with the rightful warnings if required. This test should particularly focus on the modifications of the tested release breaking the backward compatibility.
ROS controllers Test the distributed ROS simulations located in projects/languages/ros. The instructions are available at https://www.cyberbotics.com/doc/guide/tutorial-8-using-ros
Guided Tour Check that all the worlds of the guided tour behave as expected.
DavidMansolino commented 4 years ago

Here is a first draft of the table (I only restored what was in our previous quality assurance test without any update or correction). Maybe we can put this in an issue template so that we can keep it up to date and review it what do you think?

fabienrohrer commented 4 years ago

This seems good to me as-is.

stefaniapedrazzi commented 4 years ago

For the linux tests I would make this:

DavidMansolino commented 4 years ago

I updated the list taking this and the platform seetings from the previous QA into account.

stefaniapedrazzi commented 4 years ago

Thank you

stefaniapedrazzi commented 4 years ago

We forgot that on Linux we have to test both on Ubuntu 16.04 and Ubuntu 18.04. I can update the table and add different check boxes for the two versions of Ubuntu.

fabienrohrer commented 4 years ago

My tests on macOS went very well.

DavidMansolino commented 4 years ago

On windows too for me

fabienrohrer commented 4 years ago

Tests on Ubuntu 16 (Debian and tarball) went well too.

michou214 commented 4 years ago

I also try all the tests on my Windows computer. I found minor issues. 1) Tests not tested: Support UTF-8 characters, e-puck remote control, Binary compilation, DARwIn-OP, Streaming server, Extern controllers, SUMO interface, OSM importer and SUMO exporter, ROS controllers

2) projects/samples/devices/spherical_camera.wbt: The camera view is not a sphere as you can see in the following screenshot but it's possible that it's due to my graphic card. spherical_camera Configuration OpenGL: System: Windows 10 64-bit OpenGL vendor : Intel OpenGL renderer : Intel(R) HD Graphics 620 OpenGL version : 3.3.0 - Build 23.20.16.4973 Available GPU memory : N/A

3) projects/robots/softbank/nao/worlds/nao_demo.wbt: I will make a PR to correct the text displayed in the console that doesn't correspond to the movement realized by the Nao. Instead of making one step forward/backward, he makes more. Update: #1208 is ready

4) projects/languages/matlabs/worlds/e-puck_matlab.wbt: It launches Matlab but this latter get freeze. Webots is still working but stay in "Paused" mode because it wait on Matlab.

DavidMansolino commented 4 years ago

@michou214 thank you for the tests! I tested on Windows on my computer and spherical_camera.wbt is working (so we probably have an issue with your GPU). And about Matlab, I was suspecting that this was broken since a while, but it is good to have the confirmation.