Debug test failures on master branch

opendr-eu / opendr

A modular, open and non-proprietary toolkit for core robotic functionalities by harnessing deep learning

Apache License 2.0

614 stars 95 forks source link

Debug test failures on master branch #431

Closed stefaniapedrazzi closed 1 year ago

stefaniapedrazzi commented 1 year ago

Disable mobile manipulation test
PyTorch version bump
Fixed ONNX and loading interface changes
Fixed torchvision arguments
Removed inplace operation from edgespeechnets
Wheel version downgrade due to bug in the recent version
Added missing dependencies
Updated DCN loading to be backward compatible
Update macos testing evironment from deprecated macos-10.15 to macos-11

stefaniapedrazzi commented 1 year ago

It seems there are issues with the version of the installed pytorch. But checking the logs, the expected version seems to be installed:

2023-05-02T08:45:22.1287225Z Collecting torch==1.9.0 (from opendr-toolkit-engine)
2023-05-02T08:45:22.1361857Z   Downloading torch-1.9.0-cp38-cp38-manylinux1_x86_64.whl (831.4 MB)

All the failing tests are reporting error related to it. Here are some examples of the test errors:

RuntimeError: Legacy torch==2.0.0+cu117 detected, some modules may not work properly. torch>=1.4.0,<2.0.0 is required. You can use pip or conda to upgrade
AttributeError: module 'torch' has no attribute '_six'
TypeError: export() got an unexpected keyword argument 'enable_onnx_checker'

passalis commented 1 year ago

This is very weird, especially given that this runs on CI without GPU (the specific torch version has been compiled against cu117, which shouldn't be installed in any case). I suspect that there is a recent update (either in our codebase or in a dependency) that brought this as a dependency. I will try to check asap.

passalis commented 1 year ago

I think should be ok now. The way ROS starts in CI is a bit strange (through mobile_manipulations, but since it affects only CI and it works we can perhaps keep it as it is for now). I have added test_release flag to also check the generated packages.

passalis commented 1 year ago

The test wheel failures are probably due to not bumping the version tag and pulling the existing (already broken) packages from pip repo. After merging, we can start a new release with increased version number and check there if we have any other failure.

stefaniapedrazzi commented 1 year ago

It seems the manipulation_test is still failing. We can comment it for now, merge this PR, and then debug it in a separate PR so that at least we can restore the tests.

stefaniapedrazzi commented 1 year ago

The test-sources (macos-10.15) test is hanging in all the branches (including the scheduled tests). It seems that the macos-10.15 image has been deprecated one year ago and completely removed recently.

I tried to simply upgrade the version in this PR. If it shouldn't work I would merge this PR as-is and fix it in a separate branch.

stefaniapedrazzi commented 1 year ago

The test-sources is now passing both on ubuntu and macos-11.

@passalis could you please change the merge requirements and replace the test-sources (macos-10.15) with the new test-sources (macos-11)?

stefaniapedrazzi commented 1 year ago

@tsampazk oops, I change the test_suite_develop just after your approval

passalis commented 1 year ago

The test-sources is now passing both on ubuntu and macos-11.

@passalis could you please change the merge requirements and replace the test-sources (macos-10.15) with the new test-sources (macos-11)?

Done. Thanks!