faradayio / geocode-csv

Geocode a CSV file using the SmartyStreets API
7 stars 3 forks source link

"prime" libpostal library when running in server mode #16

Closed tomplex closed 1 year ago

tomplex commented 1 year ago

Libpostal depends on a language model which takes 5-10 seconds to load into memory. Currently, that happens at the time of the first request; this PR instead causes it to happen as part of application startup when it is run in server mode.

$ export RUST_LOG=debug
$ cargo build && cargo run -- --spec fake_spec.json --geocoder libpostal server
    Finished dev [unoptimized + debuginfo] target(s) in 0.10s
    Finished dev [unoptimized + debuginfo] target(s) in 0.10s
     Running `target/debug/geocode-csv --spec fake_spec.json --geocoder libpostal server`
2023-08-09T23:53:24.499782Z  INFO geocode-csv: geocode_csv: new
2023-08-09T23:53:24.499821Z DEBUG geocode-csv: geocode_csv: geocode-csv 1.3.9
2023-08-09T23:53:24.501108Z DEBUG geocode-csv:libpostal_setup_datadir: libpostal_rust::init: new
2023-08-09T23:53:24.501136Z DEBUG geocode-csv:libpostal_setup_datadir: libpostal_rust::probe: probing for libpostal data directory
2023-08-09T23:53:24.501159Z DEBUG geocode-csv:libpostal_setup_datadir: libpostal_rust::probe: using /usr/local/share/libpostal as libpostal data directory
2023-08-09T23:53:24.567769Z DEBUG geocode-csv:libpostal_setup_datadir: libpostal_rust::init: close time.busy=66.6ms time.idle=30.8µs
2023-08-09T23:53:24.567855Z DEBUG geocode-csv:libpostal_setup_parser_datadir: libpostal_rust::init: new
2023-08-09T23:53:28.198433Z DEBUG geocode-csv:libpostal_setup_parser_datadir: libpostal_rust::init: close time.busy=3.63s time.idle=26.3µs