Importing PROTOs from non-official web sources

gabryelreyes commented 3 months ago

Describe the Bug

When using EXTERNPROTO to import a PROTO file such as this one into a world, the following message is printed to the console:

ERROR: The cascaded URL inferring mechanism is supported only for official Webots assets.
ERROR: 'https://raw.githubusercontent.com/BlueAndi/RadonUlzer/main/webots/protos/Zumo32U4.proto':115:22: error: Skipped unknown 'Rubber' node or PROTO.

In this case, a non-official (custom) PROTO is being loaded, which also has its own dependencies (in this case, the official Rubber PROTO, but it may as well be other custom PROTO).

I traced the log message to WbProtoManager.cpp which determines that the URL is not from the official repository by using a RegEx expression from WbUrl.cpp.

Could you please shed some light onto the reason behind only supporting official Webots assets in this feature?

Additional context

We are working on the RadonUlzer project, which now has evolved to support both real hardware and simulated hardware using the same application code. This is achieved by separation of the HALs into different repositories. For example, for Webots there is the ZumoHALWebots. This HAL is dependent on the Zumo32U4 PROTO, which currently is found in another repository together with the wbt files.

The main concern is mismatching configurations between the PROTO and the corresponding HAL and drivers. To prevent this the optimal option is to get the PROTO file directly from the GitHub repository based on the HAL that is being processed by the build system. We would also like to avoid manually copying the PROTO file each time from the HAL Repository to the Worlds folder for the same risk of outdated/mismatching configurations after an update.

Steps to Reproduce

Create a world.
Add line EXTERNPROTO "https://raw.githubusercontent.com/BlueAndi/RadonUlzer/main/webots/protos/Zumo32U4.proto"
Open wbt file.
See message in the console.

Example File:

#VRML_SIM R2023b utf8
EXTERNPROTO "https://raw.githubusercontent.com/cyberbotics/webots/R2023b/projects/objects/backgrounds/protos/TexturedBackground.proto"
EXTERNPROTO "https://raw.githubusercontent.com/cyberbotics/webots/R2023b/projects/objects/backgrounds/protos/TexturedBackgroundLight.proto"
EXTERNPROTO "https://raw.githubusercontent.com/cyberbotics/webots/R2023b/projects/objects/floors/protos/RectangleArena.proto"
EXTERNPROTO "https://raw.githubusercontent.com/BlueAndi/RadonUlzer/main/webots/protos/Zumo32U4.proto"

WorldInfo {
}
Viewpoint {
  orientation -0.5773 0.5773 0.5773 2.0944
  position 0 0 10
}
TexturedBackground {
}
TexturedBackgroundLight {
}
RectangleArena {
}
Zumo32U4 {
  translation 0 0 0
  rotation 0 1 0 0
  name "Zumo"
}

Expected behavior

Loading the PROTO from a web URL with all its dependencies.

System

Operating System: Windows 11
Graphics Card: Nvidia GeForce Mx550

omichel commented 3 months ago

I believe this restriction is simply due to a lack of implementation. If you believe you could implement it by yourself, please go ahead and propose a patch in a new Pull Request. We will review it and likely approve it. I just added you in the group of the Webots committers, so that you won't have to fork this repository to propose a PR.

gabryelreyes commented 3 months ago

Thank you @omichel for the invitation. I will take a look and try to come up with an implementation if I find the time.

gabryelreyes commented 2 months ago

I am focusing on the path-infering block.

The first if checks if the asset is web and cached, while the second block checks if its local, or relative (which also accepts web URLs), so in this case we go into the second block. The objective of the second block is then to "build" a new URL based on the parent's URL prefix, However this is not required when the asset is web! The address remains unchanged and it is downloaded from whatever address is supplied.

I believe there is a third case, which is not handled here, and that is when the asset is web and not cached. If the first block is slightly modified, the web URL remains untouched.

if (WbUrl::isWeb(protoDeclaration))
{
  if (WbNetwork::instance()->isCachedWithMapUpdate(modelPath))
  modelPath = protoDeclaration;
}

RESULT: The first time the world loads, the rubber texture is not displayed, however it was succesfully imported. Once the world is reloaded, the texture is found in the cache and is correctly displayed. This means for me, that the paths are correct and the file is correctly downloaded and added to the model list.

Maybe anyone has an idea, why the texture is only shown after the reload? Am I missing something?

gabryelreyes commented 2 months ago

The texture not being loaded is a problem existing in the latest release as well, and its not product of my change in the code

cyberbotics / webots