Proxy support for database

[Skimige]

2024-06-21T14:31:22.972862Z  INFO config_map: reliquary_archiver::export::fribbels: initializing database from online sources, this might take a while...
The application panicked (crashed).
Message:  called `Result::unwrap()` on an `Err` value: Transport(Transport { kind: Dns, message: Some("resolve dns name 'raw.githubusercontent.com:443'"), url: Some(Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("raw.githubusercontent.com")), port: None, path: "/Dimbreath/StarRailData/master/ExcelOutput/AvatarConfig.json", query: None, fragment: None }), source: Some(Os { code: 11004, kind: Uncategorized, message: "请求的名称有效，但是找不到请求的类型的数据。" }) })
Location: src\export\fribbels.rs:300

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
Warning: SpanTrace capture is Unsupported.
Ensure that you've setup a tracing-error ErrorLayer and the semver versions are compatible

This issue looks like a duplicate of #13, lol.

I am experiencing difficulty connecting to raw.githubusercontent.com from my home network. Despite setting the http_proxy and all_proxy environment variables, the connection does not work, and I cannot see any log entries from my proxy.

Based on my research, ureq should automatically support proxy environments, but this doesn't seem to be the case here.

As a workaround, I replaced the link with a mirror (built using actions in a private fork), and it worked.

Given that a local database is not the preferred solution (#11), is there any possibility of implementing proxy support for database fetching?

Thank you for developing this tool!

[IceDynamix]

i think i will ship the database with future releases. the local db pr hasn't been merged yet because i haven't gotten around to testing it

[IDwrong]

Inspired by @Skimige , I searched for "ureq proxy" and found this. It mentions that

I see that wasm-pack and binary-install all calls ureq::get(..) to download, which initializes an agent with the default setting AgentBuilder::new().build(), and by default try_proxy_from_env of AgentBuilder is set to false, so it ended up not using the proxy environment variables.

I studied the Rust documentation and tried to modify get() so that it can use the proxy from the HTTP_HOSTS environment variable.

fn get<T: DeserializeOwned>(url: String) -> T {
    debug!(url, "requesting from resource");
    ureq::AgentBuilder::new().try_proxy_from_env(true).build()
        .get(&url)
        .call()
        .unwrap()
        .into_json()
        .unwrap()
}

I'm glad it works well.

However, I use an app which controls the 'manual proxy setup' settings in Windows, and it uses the registry. So I studied it further and modified get() to use the proxy configured in the registry keys.

fn get<T: DeserializeOwned>(url: String) -> T {
    debug!(url, "requesting from resource");
    // Open registry key.
    let hkcu = RegKey::predef(HKEY_CURRENT_USER);
    let internet_settings = hkcu.open_subkey_with_flags(
        "Software\\Microsoft\\Windows\\CurrentVersion\\Internet Settings",
        KEY_READ,
    ).unwrap();
    // Read Proxy Settings.
    let proxy_enable: u32 = internet_settings.get_value("ProxyEnable").unwrap();
    let proxy_server: String = internet_settings.get_value("ProxyServer").unwrap();

    if proxy_enable == 0 {
        ureq::get(&url)
            .call()
            .unwrap()
            .into_json()
            .unwrap()
    } else {
        ureq::AgentBuilder::new()
            .proxy(ureq::Proxy::new(proxy_server).unwrap())
            .build()
            .get(&url)
            .call()
            .unwrap()
            .into_json()
            .unwrap()
    }
}

Without a proxy, it's only 4KB/s when updating Database. But now I can have a normal download speed. ^ ^

Of course, modifying get() directly is not very elegant. Perhaps the logic for creating agent could be moved to a new struct or main.rs, and then agent could be passed to Database for use.