gdesmar
commented
3 months ago The URLDownloader service can be configured to use many proxies (or not) and allow the user to select one or another. If you want to force a proxy, you can also have a single entry in the list of choices, and that will make it mandatory. We have situations where we have multiple proxies and a choice for the user to make, and those proxies could even be different from the proxy used by the rest of the system.
You can configure the URLDownloader service by going to the list of services, and clicking on URLDownloader (or going to /admin/services/URLDownloader directly). You should find a tab named PARAMETERS. Two parameters are important, is it easier to understand it by starting with the proxies [json] under the service variables toward the bottom of the page. You can create new entries based on the following pattern:
The UI to edit json in this box takes a second to figure out: by hovering on the json, you should see a blue plus sign appear toward the top:
This will allow you to create a new key. By hovering on the new key, you should be able to edit it:
You can then type in
{} and click on the new dictionary button (bottom right in the next screenshot):
You should now be able to add two new keys, for
http and https.
Side note: The code should handles having a single entry with the following, and will use it for all schemes (http/https):
After configuring the service proxies, you can look toward the top, under User Specified Parameters, there should be proxy [list].
You will be able to add the name of the key you added (
my_new_proxy in my example) so that the users can select it.
The entry with a star is going to be the default selection if a user does not configure it. You can delete all other entries from here to force a single one.
If you can test it out and make sure it does as you would expect, I will gladly use this issue to add that information to URLDownloader's README.
precise0
Hello, I am currently running the Dockerized full appliance deployment of Assemblyline 4 deployed using the appliance deployment instructions for RHEL systems.
All the features work well however I have an issue with getting the HTTP proxy settings to apply properly to the URLDownloader service. I have tried two methods, first I used the config.yml ui -> url_submission_proxies parameter and I tried a few formats for adding the proxy in such as a dictionary keyed on the normal environment variables for HTTP proxies as well as just simply specifying one parameter "proxy" with my selected proxy.
The second method I used was applying a proxy in the Assemblyline WebUI for the URLDownloader service, the container would restart but none of my changes seemed to apply. The URLs would fail to download and timeout eventually in all cases.
In all honesty I think the problem is in the lack of documentation on this feature, to me it's unclear how to effectively declare the HTTP proxy settings in the correct format. Could the documentation please be updated to clearly describe how to apply these settings? I'd be thankful for a little guidance in this matter.
Expected behavior URLs submitted to the URLDownloader service should use the configured HTTP proxy
Environment (please complete the following information if pertinent): Assemblyline 4 Full Appliance Docker deployment 4.5.stable URLDownloader service version: 4.5.0.13