✨ Config option to use operating system certificates alongside `rustls` certificates

Work Expected From The Issue

Provide a new config option to users to enable or disable operating system certificates to be used alongside rustls certificates. Giving the user the control to whether to use operating system certificates or not while considering security in mind.

The issue expects the following files to be changed/modified:

src/results/aggregator.rs
src/config/parser.rs
websurfx/config.lua

[!Note] All the files that are expected to be changed are located under the codebase (websurfx directory).

Reason Behind These Changes

The reason behind having these changes it to give the user the control to whether to use operating system certificates or not while considering security in mind.

Sample Code

The sample codes for the files from the above mentioned files has been provided below:

aggregator.rs

//! This module provides the functionality to scrape and gathers all the results from the upstream
//! search engines and then removes duplicate results.

use super::user_agent::random_user_agent;
use crate::config::parser::Config;
use crate::handler::{file_path, FileType};
use crate::models::{
    aggregation_models::{EngineErrorInfo, SearchResult, SearchResults},
    engine_models::{EngineError, EngineHandler},
};

use error_stack::Report;
use futures::stream::FuturesUnordered;
use regex::Regex;
use reqwest::{Client, ClientBuilder};
use std::sync::Arc;
use std::time::{SystemTime, UNIX_EPOCH};
use tokio::{
    fs::File,
    io::{AsyncBufReadExt, BufReader},
    task::JoinHandle,
    time::Duration,
};

/// A constant for holding the prebuilt Client globally in the app.
static CLIENT: std::sync::OnceLock<Client> = std::sync::OnceLock::new();

/// Aliases for long type annotations

type FutureVec =
    FuturesUnordered<JoinHandle<Result<Vec<(String, SearchResult)>, Report<EngineError>>>>;

/// The function aggregates the scraped results from the user-selected upstream search engines.
/// These engines can be chosen either from the user interface (UI) or from the configuration file.
/// The code handles this process by matching the selected search engines and adding them to a vector.
/// This vector is then used to create an asynchronous task vector using `tokio::spawn`, which returns
/// a future. This future is awaited in another loop. Once the results are collected, they are filtered
/// to remove any errors and ensure only proper results are included. If an error is encountered, it is
/// sent to the UI along with the name of the engine and the type of error. This information is finally
/// placed in the returned `SearchResults` struct.
///
/// Additionally, the function eliminates duplicate results. If two results are identified as coming from
/// multiple engines, their names are combined to indicate that the results were fetched from these upstream
/// engines. After this, all the data in the `Vec` is removed and placed into a struct that contains all
/// the aggregated results in a vector. Furthermore, the query used is also added to the struct. This step is
/// necessary to ensure that the search bar in the search remains populated even when searched from the query URL.
///
/// Overall, this function serves to aggregate scraped results from user-selected search engines, handling errors,
/// removing duplicates, and organizing the data for display in the UI.
///
/// # Example:
///
/// If you search from the url like `https://127.0.0.1/search?q=huston` then the search bar should
/// contain the word huston and not remain empty.
///
/// # Arguments
///
/// * `query` - Accepts a string to query with the above upstream search engines.
/// * `page` - Accepts an u32 page number.
/// * `random_delay` - Accepts a boolean value to add a random delay before making the request.
/// * `debug` - Accepts a boolean value to enable or disable debug mode option.
/// * `upstream_search_engines` - Accepts a vector of search engine names which was selected by the
/// * `request_timeout` - Accepts a time (secs) as a value which controls the server request timeout.
/// user through the UI or the config file.
///
/// # Error
///
/// Returns an error a reqwest and scraping selector errors if any error occurs in the results
/// function in either `searx` or `duckduckgo` or both otherwise returns a `SearchResults struct`
/// containing appropriate values.
pub async fn aggregate(
    query: &str,
    page: u32,
    config: &Config,
    upstream_search_engines: &[EngineHandler],
    safe_search: u8,
) -> Result<SearchResults, Box<dyn std::error::Error>> {
    let client = CLIENT.get_or_init(|| {
        ClientBuilder::new()
            .timeout(Duration::from_secs(config.request_timeout as u64)) // Add timeout to request to avoid DDOSing the server
            .pool_idle_timeout(Duration::from_secs(
                config.pool_idle_connection_timeout as u64,
            ))
            .tcp_keepalive(Duration::from_secs(config.tcp_connection_keepalive as u64))
            .connect_timeout(Duration::from_secs(config.request_timeout as u64)) // Add timeout to request to avoid DDOSing the server
+            .use_rustls_tls()
+            .tls_built_in_root_certs(config.operating_system_tls_certificates)
            .https_only(true)
            .gzip(true)
            .brotli(true)
            .http2_adaptive_window(config.adaptive_window)
            .build()
            .unwrap()
    });

    let user_agent: &str = random_user_agent();

    // Add a random delay before making the request.
    if config.aggregator.random_delay || !config.debug {
        let nanos = SystemTime::now().duration_since(UNIX_EPOCH)?.subsec_nanos() as f32;
        let delay = ((nanos / 1_0000_0000 as f32).floor() as u64) + 1;
        tokio::time::sleep(Duration::from_secs(delay)).await;
    }

    let mut names: Vec<&str> = Vec::with_capacity(0);

    // create tasks for upstream result fetching
    let tasks: FutureVec = FutureVec::new();

    let query: Arc<String> = Arc::new(query.to_string());
    for engine_handler in upstream_search_engines {
        let (name, search_engine) = engine_handler.clone().into_name_engine();
        names.push(name);
        let query_partially_cloned = query.clone();
        tasks.push(tokio::spawn(async move {
            search_engine
                .results(
                    &query_partially_cloned,
                    page,
                    user_agent,
                    client,
                    safe_search,
                )
                .await
        }));
    }

    // get upstream responses
    let mut responses = Vec::with_capacity(tasks.len());

    for task in tasks {
        if let Ok(result) = task.await {
            responses.push(result)
        }
    }

    // aggregate search results, removing duplicates and handling errors the upstream engines returned
    let mut result_map: Vec<(String, SearchResult)> = Vec::new();
    let mut engine_errors_info: Vec<EngineErrorInfo> = Vec::new();

    let mut handle_error = |error: &Report<EngineError>, engine_name: &'static str| {
        log::error!("Engine Error: {:?}", error);
        engine_errors_info.push(EngineErrorInfo::new(
            error.downcast_ref::<EngineError>().unwrap(),
            engine_name,
        ));
    };

    for _ in 0..responses.len() {
        let response = responses.pop().unwrap();
        let engine = names.pop().unwrap();

        if result_map.is_empty() {
            match response {
                Ok(results) => result_map = results,
                Err(error) => handle_error(&error, engine),
            };
            continue;
        }

        match response {
            Ok(result) => {
                result.into_iter().for_each(|(key, value)| {
                    match result_map.iter().find(|(key_s, _)| key_s == &key) {
                        Some(value) => value.1.to_owned().add_engines(engine),
                        None => result_map.push((key, value)),
                    };
                });
            }
            Err(error) => handle_error(&error, engine),
        };
    }

    if safe_search >= 3 {
        let mut blacklist_map: Vec<(String, SearchResult)> = Vec::new();
        filter_with_lists(
            &mut result_map,
            &mut blacklist_map,
            file_path(FileType::BlockList)?,
        )
        .await?;

        filter_with_lists(
            &mut blacklist_map,
            &mut result_map,
            file_path(FileType::AllowList)?,
        )
        .await?;

        drop(blacklist_map);
    }

    let mut results: Vec<SearchResult> = result_map
        .iter()
        .map(|(_, value)| {
            let mut copy = value.clone();
            if !copy.url.contains("temu.com") {
                copy.calculate_relevance(query.as_str())
            }
            copy
        })
        .collect();
    sort_search_results(&mut results);

    Ok(SearchResults::new(results, &engine_errors_info))
}

/// Filters a map of search results using a list of regex patterns.
///
/// # Arguments
///
/// * `map_to_be_filtered` - A mutable reference to a `Vec` of search results to filter, where the filtered results will be removed from.
/// * `resultant_map` - A mutable reference to a `Vec` to hold the filtered results.
/// * `file_path` - A `&str` representing the path to a file containing regex patterns to use for filtering.
///
/// # Errors
///
/// Returns an error if the file at `file_path` cannot be opened or read, or if a regex pattern is invalid.
pub async fn filter_with_lists(
    map_to_be_filtered: &mut Vec<(String, SearchResult)>,
    resultant_map: &mut Vec<(String, SearchResult)>,
    file_path: &str,
) -> Result<(), Box<dyn std::error::Error>> {
    let reader = BufReader::new(File::open(file_path).await?);
    let mut lines = reader.lines();

    while let Some(line) = lines.next_line().await? {
        let re = Regex::new(line.trim())?;

        let mut length = map_to_be_filtered.len();
        let mut idx: usize = Default::default();
        // Iterate over each search result in the map and check if it matches the regex pattern
        while idx < length {
            let ele = &map_to_be_filtered[idx];
            let ele_inner = &ele.1;
            match re.is_match(&ele.0.to_lowercase())
                || re.is_match(&ele_inner.title.to_lowercase())
                || re.is_match(&ele_inner.description.to_lowercase())
            {
                true => {
                    // If the search result matches the regex pattern, move it from the original map to the resultant map
                    resultant_map.push(map_to_be_filtered.swap_remove(idx));
                    length -= 1;
                }
                false => idx += 1,
            };
        }
    }

    Ok(())
}
/// Sorts  SearchResults by relevance score.
/// <br> sort_unstable is used as its faster,stability is not an issue on our side.
/// For reasons why, check out [`this`](https://rust-lang.github.io/rfcs/1884-unstable-sort.html)
///  # Arguments
///  * `results` - A mutable slice or Vec of SearchResults
///  
fn sort_search_results(results: &mut [SearchResult]) {
    results.sort_unstable_by(|a, b| {
        use std::cmp::Ordering;

        b.relevance_score
            .partial_cmp(&a.relevance_score)
            .unwrap_or(Ordering::Less)
    })
}
#[cfg(test)]
mod tests {
    use super::*;
    use smallvec::smallvec;
    use std::io::Write;
    use tempfile::NamedTempFile;

    #[tokio::test]
    async fn test_filter_with_lists() -> Result<(), Box<dyn std::error::Error>> {
        // Create a map of search results to filter
        let mut map_to_be_filtered = Vec::new();
        map_to_be_filtered.push((
            "https://www.example.com".to_owned(),
            SearchResult {
                title: "Example Domain".to_owned(),
                url: "https://www.example.com".to_owned(),
                description: "This domain is for use in illustrative examples in documents."
                    .to_owned(),
                relevance_score: 0.0,
                engine: smallvec!["Google".to_owned(), "Bing".to_owned()],
            },
        ));
        map_to_be_filtered.push((
            "https://www.rust-lang.org/".to_owned(),
            SearchResult {
                title: "Rust Programming Language".to_owned(),
                url: "https://www.rust-lang.org/".to_owned(),
                description: "A systems programming language that runs blazingly fast, prevents segfaults, and guarantees thread safety.".to_owned(),
                engine: smallvec!["Google".to_owned(), "DuckDuckGo".to_owned()],
                relevance_score:0.0
            },)
        );

        // Create a temporary file with regex patterns
        let mut file = NamedTempFile::new()?;
        writeln!(file, "example")?;
        writeln!(file, "rust")?;
        file.flush()?;

        let mut resultant_map = Vec::new();
        filter_with_lists(
            &mut map_to_be_filtered,
            &mut resultant_map,
            file.path().to_str().unwrap(),
        )
        .await?;

        assert_eq!(resultant_map.len(), 2);
        assert!(resultant_map
            .iter()
            .any(|(key, _)| key == "https://www.example.com"));
        assert!(resultant_map
            .iter()
            .any(|(key, _)| key == "https://www.rust-lang.org/"));
        assert_eq!(map_to_be_filtered.len(), 0);

        Ok(())
    }

    #[tokio::test]
    async fn test_filter_with_lists_wildcard() -> Result<(), Box<dyn std::error::Error>> {
        let mut map_to_be_filtered = Vec::new();
        map_to_be_filtered.push((
            "https://www.example.com".to_owned(),
            SearchResult {
                title: "Example Domain".to_owned(),
                url: "https://www.example.com".to_owned(),
                description: "This domain is for use in illustrative examples in documents."
                    .to_owned(),
                engine: smallvec!["Google".to_owned(), "Bing".to_owned()],
                relevance_score: 0.0,
            },
        ));
        map_to_be_filtered.push((
            "https://www.rust-lang.org/".to_owned(),
            SearchResult {
                title: "Rust Programming Language".to_owned(),
                url: "https://www.rust-lang.org/".to_owned(),
                description: "A systems programming language that runs blazingly fast, prevents segfaults, and guarantees thread safety.".to_owned(),
                engine: smallvec!["Google".to_owned(), "DuckDuckGo".to_owned()],
                relevance_score:0.0
            },
        ));

        // Create a temporary file with a regex pattern containing a wildcard
        let mut file = NamedTempFile::new()?;
        writeln!(file, "ex.*le")?;
        file.flush()?;

        let mut resultant_map = Vec::new();

        filter_with_lists(
            &mut map_to_be_filtered,
            &mut resultant_map,
            file.path().to_str().unwrap(),
        )
        .await?;

        assert_eq!(resultant_map.len(), 1);
        assert!(resultant_map
            .iter()
            .any(|(key, _)| key == "https://www.example.com"));
        assert_eq!(map_to_be_filtered.len(), 1);
        assert!(map_to_be_filtered
            .iter()
            .any(|(key, _)| key == "https://www.rust-lang.org/"));

        Ok(())
    }

    #[tokio::test]
    async fn test_filter_with_lists_file_not_found() {
        let mut map_to_be_filtered = Vec::new();

        let mut resultant_map = Vec::new();

        // Call the `filter_with_lists` function with a non-existent file path
        let result = filter_with_lists(
            &mut map_to_be_filtered,
            &mut resultant_map,
            "non-existent-file.txt",
        );

        assert!(result.await.is_err());
    }

    #[tokio::test]
    async fn test_filter_with_lists_invalid_regex() {
        let mut map_to_be_filtered = Vec::new();
        map_to_be_filtered.push((
            "https://www.example.com".to_owned(),
            SearchResult {
                title: "Example Domain".to_owned(),
                url: "https://www.example.com".to_owned(),
                description: "This domain is for use in illustrative examples in documents."
                    .to_owned(),
                engine: smallvec!["Google".to_owned(), "Bing".to_owned()],
                relevance_score: 0.0,
            },
        ));

        let mut resultant_map = Vec::new();

        // Create a temporary file with an invalid regex pattern
        let mut file = NamedTempFile::new().unwrap();
        writeln!(file, "example(").unwrap();
        file.flush().unwrap();

        let result = filter_with_lists(
            &mut map_to_be_filtered,
            &mut resultant_map,
            file.path().to_str().unwrap(),
        );

        assert!(result.await.is_err());
    }
}

config.lua

 -- ### General ###
logging = true -- an option to enable or disable logs.
debug = false -- an option to enable or disable debug mode.
threads = 10 -- the amount of threads that the app will use to run (the value should be greater than 0).

 -- ### Server ###
port = "8080" -- port on which server should be launched
binding_ip = "127.0.0.1" --ip address on the which server should be launched.
production_use = false -- whether to use production mode or not (in other words this option should be used if it is to be used to host it on the server to provide a service to a large number of users (more than one))
 -- if production_use is set to true
 -- There will be a random delay before sending the request to the search engines, this is to prevent DDoSing the upstream search engines from a large number of simultaneous requests.
request_timeout = 30 -- timeout for the search requests sent to the upstream search engines to be fetched (value in seconds).
tcp_connection_keepalive = 30 -- the amount of time the tcp connection should remain alive (or connected to the server). (value in seconds).
pool_idle_connection_timeout = 30 -- timeout for the idle connections in the reqwest HTTP connection pool (value in seconds).
rate_limiter = {
    number_of_requests = 20, -- The number of request that are allowed within a provided time limit.
    time_limit = 3, -- The time limit in which the quantity of requests that should be accepted.
}
https_adaptive_window_size = false -- Set whether the server will use an adaptive/dynamic HTTPS window size, see https://httpwg.org/specs/rfc9113.html#fc-principles

+ operating_system_tls_certificates = true -- Set whether the server will use operating system's tls certificates alongside rustls certificates while fetching search results from the upstream engines.

 -- ### Search ###
 -- Filter results based on different levels. The levels provided are:
 -- {{
 -- 0 - None
 -- 1 - Low
 -- 2 - Moderate
 -- 3 - High
 -- 4 - Aggressive
 -- }}
safe_search = 2

 -- ### Website ###
 -- The different colorschemes provided are:
 -- {{
 -- catppuccin-mocha
 -- dark-chocolate
 -- dracula
 -- gruvbox-dark
 -- monokai
 -- nord
 -- oceanic-next
 -- one-dark
 -- solarized-dark
 -- solarized-light
 -- tokyo-night
 -- tomorrow-night
 -- }}
colorscheme = "catppuccin-mocha" -- the colorscheme name which should be used for the website theme
 -- The different themes provided are:
 -- {{
 -- simple
 -- }}
theme = "simple" -- the theme name which should be used for the website
 -- The different animations provided are:
 -- {{
 -- simple-frosted-glow
 -- }}
animation = "simple-frosted-glow" -- the animation name which should be used with the theme or `nil` if you don't want any animations.

 -- ### Caching ###
redis_url = "redis://127.0.0.1:8082" -- redis connection url address on which the client should connect on.
cache_expiry_time = 600 -- This option takes the expiry time of the search results (value in seconds and the value should be greater than or equal to 60 seconds).

 -- ### Search Engines ###
upstream_search_engines = {
    DuckDuckGo = true,
    Searx = false,
    Brave = false,
    Startpage = false,
    LibreX = false,
    Mojeek = false,
    Bing = false,
} -- select the upstream search engines from which the results should be fetched.

parser.rs

//! This module provides the functionality to parse the lua config and convert the config options
//! into rust readable form.

use crate::handler::{file_path, FileType};

use crate::models::parser_models::{AggregatorConfig, RateLimiter, Style};
use log::LevelFilter;
use mlua::Lua;
use std::{collections::HashMap, fs, thread::available_parallelism};

/// A named struct which stores the parsed config file options.
pub struct Config {
    /// It stores the parsed port number option on which the server should launch.
    pub port: u16,
    /// It stores the parsed ip address option on which the server should launch
    pub binding_ip: String,
    /// It stores the theming options for the website.
    pub style: Style,
    #[cfg(feature = "redis-cache")]
    /// It stores the redis connection url address on which the redis
    /// client should connect.
    pub redis_url: String,
    #[cfg(any(feature = "redis-cache", feature = "memory-cache"))]
    /// It stores the max TTL for search results in cache.
    pub cache_expiry_time: u16,
    /// It stores the option to whether enable or disable production use.
    pub aggregator: AggregatorConfig,
    /// It stores the option to whether enable or disable logs.
    pub logging: bool,
    /// It stores the option to whether enable or disable debug mode.
    pub debug: bool,
    /// It toggles whether to use adaptive HTTP windows
    pub adaptive_window: bool,
    /// It stores all the engine names that were enabled by the user.
    pub upstream_search_engines: HashMap<String, bool>,
    /// It stores the time (secs) which controls the server request timeout.
    pub request_timeout: u8,
    /// It stores the number of threads which controls the app will use to run.
    pub threads: u8,
    /// It stores configuration options for the ratelimiting middleware.
    pub rate_limiter: RateLimiter,
    /// It stores the level of safe search to be used for restricting content in the
    /// search results.
    pub safe_search: u8,
    /// It stores the TCP connection keepalive duration in seconds.
    pub tcp_connection_keepalive: u8,
    /// It stores the pool idle connection timeout in seconds.
    pub pool_idle_connection_timeout: u8,
+    pub operating_system_tls_certificates: bool,
}

impl Config {
    /// A function which parses the config.lua file and puts all the parsed options in the newly
    /// constructed Config struct and returns it.
    ///
    /// # Arguments
    ///
    /// * `logging_initialized` - It takes a boolean which ensures that the logging doesn't get
    /// initialized twice. Pass false if the logger has not yet been initialized.
    ///
    /// # Error
    ///
    /// Returns a lua parse error if parsing of the config.lua file fails or has a syntax error
    /// or io error if the config.lua file doesn't exists otherwise it returns a newly constructed
    /// Config struct with all the parsed config options from the parsed config file.
    pub fn parse(logging_initialized: bool) -> Result<Self, Box<dyn std::error::Error>> {
        let lua = Lua::new();
        let globals = lua.globals();

        lua.load(&fs::read_to_string(file_path(FileType::Config)?)?)
            .exec()?;

        let parsed_threads: u8 = globals.get::<_, u8>("threads")?;

        let debug: bool = globals.get::<_, bool>("debug")?;
        let logging: bool = globals.get::<_, bool>("logging")?;
        let adaptive_window: bool = globals.get::<_, bool>("adaptive_window")?;

        if !logging_initialized {
            set_logging_level(debug, logging);
        }

        let threads: u8 = if parsed_threads == 0 {
            let total_num_of_threads: usize = available_parallelism()?.get() / 2;
            log::error!(
                "Config Error: The value of `threads` option should be a non zero positive integer"
            );
            log::error!("Falling back to using {} threads", total_num_of_threads);
            total_num_of_threads as u8
        } else {
            parsed_threads
        };

        let rate_limiter = globals.get::<_, HashMap<String, u8>>("rate_limiter")?;

        let parsed_safe_search: u8 = globals.get::<_, u8>("safe_search")?;
        let safe_search: u8 = match parsed_safe_search {
            0..=4 => parsed_safe_search,
            _ => {
                log::error!("Config Error: The value of `safe_search` option should be a non zero positive integer from 0 to 4.");
                log::error!("Falling back to using the value `1` for the option");
                1
            }
        };

        #[cfg(any(feature = "redis-cache", feature = "memory-cache"))]
        let parsed_cet = globals.get::<_, u16>("cache_expiry_time")?;
        #[cfg(any(feature = "redis-cache", feature = "memory-cache"))]
        let cache_expiry_time = match parsed_cet {
            0..=59 => {
                log::error!(
                    "Config Error: The value of `cache_expiry_time` must be greater than 60"
                );
                log::error!("Falling back to using the value `60` for the option");
                60
            }
            _ => parsed_cet,
        };

        Ok(Config {
            port: globals.get::<_, u16>("port")?,
            binding_ip: globals.get::<_, String>("binding_ip")?,
            style: Style::new(
                globals.get::<_, String>("theme")?,
                globals.get::<_, String>("colorscheme")?,
                globals.get::<_, Option<String>>("animation")?,
            ),
            #[cfg(feature = "redis-cache")]
            redis_url: globals.get::<_, String>("redis_url")?,
            aggregator: AggregatorConfig {
                random_delay: globals.get::<_, bool>("production_use")?,
            },
            logging,
            debug,
            adaptive_window,
            upstream_search_engines: globals
                .get::<_, HashMap<String, bool>>("upstream_search_engines")?,
            request_timeout: globals.get::<_, u8>("request_timeout")?,
            tcp_connection_keepalive: globals.get::<_, u8>("tcp_connection_keepalive")?,
            pool_idle_connection_timeout: globals.get::<_, u8>("pool_idle_connection_timeout")?,
+            operating_system_tls_certificates: globals.get::<_, bool>("operating_system_tls_certificates")?,
            threads,
            rate_limiter: RateLimiter {
                number_of_requests: rate_limiter["number_of_requests"],
                time_limit: rate_limiter["time_limit"],
            },
            safe_search,
            #[cfg(any(feature = "redis-cache", feature = "memory-cache"))]
            cache_expiry_time,
        })
    }
}

/// a helper function that sets the proper logging level
///
/// # Arguments
///
/// * `debug` - It takes the option to whether enable or disable debug mode.
/// * `logging` - It takes the option to whether enable or disable logs.
fn set_logging_level(debug: bool, logging: bool) {
    if let Ok(pkg_env_var) = std::env::var("PKG_ENV") {
        if pkg_env_var.to_lowercase() == "dev" {
            env_logger::Builder::new()
                .filter(None, LevelFilter::Trace)
                .init();
            return;
        }
    }

    // Initializing logging middleware with level set to default or info.
    let log_level = match (debug, logging) {
        (true, true) => LevelFilter::Debug,
        (true, false) => LevelFilter::Debug,
        (false, true) => LevelFilter::Info,
        (false, false) => LevelFilter::Error,
    };

    env_logger::Builder::new().filter(None, log_level).init();
}

[!Note]

To get started contributing make sure to read the contributing.md file for the guidlines on how to contribute in this project

To contribute first fork this project by following this video tutorial if you are not familliar with process and add your changes and make a pull request with the changes to this repository and if you are new to GitHub then follow this video tutorial to get started contributing :slightly_smiling_face: .

neon-mmd / websurfx