When working with FTP in Funnel, we need to supply FTP credentials through the URLs. In the current implementation, input and TES URIs/URLs are parsed with urllib.parse.urlparse. Out of the resulting fragments, the netloc is then passed to socket.gethostbyname to get the IP of that host. However, the netloc extracted via urllib.parse.urlparse retains (basic) authorization credentials (e.g., user:password@host.name), but socket.gethostbyname is not able to parse these and throws a socket.gaierror. This exception is caught, but is handled in such a way that the list of input URIs is either incomplete or empty. In the latter case, this leads to an error in pro_tes.middleware.task_distribution.distance.task_distribution because no TES/input URL/URI combinations can be compiled (a situtation that is not handled).
To address this issue fully, the following should be done:
[x] Remove auth credentials from URLs/URIs before determining IPs
[x] Return a custom error amounting to 400 if input URIs cannot be parsed gethostbyname
[x] Return a custom error amounting to 500 if TES URIs cannot be parsed by gethostbyname
[x] Add comprehensive type hints to module
[x] The distance-based logic should not fail if, for some reason, no TES/input IP combinations can be constructed; in that case, proTES should fall back to random distribution
[x] Write unit tests for all code in the module and run in the CI
When working with FTP in Funnel, we need to supply FTP credentials through the URLs. In the current implementation, input and TES URIs/URLs are parsed with
urllib.parse.urlparse. Out of the resulting fragments, thenetlocis then passed tosocket.gethostbynameto get the IP of that host. However, thenetlocextracted viaurllib.parse.urlparseretains (basic) authorization credentials (e.g.,user:password@host.name), butsocket.gethostbynameis not able to parse these and throws asocket.gaierror. This exception is caught, but is handled in such a way that the list of input URIs is either incomplete or empty. In the latter case, this leads to an error inpro_tes.middleware.task_distribution.distance.task_distributionbecause no TES/input URL/URI combinations can be compiled (a situtation that is not handled).To address this issue fully, the following should be done:
400if input URIs cannot be parsedgethostbyname500if TES URIs cannot be parsed bygethostbyname