search

triton-inference-server / fil_backend

FIL backend for the Triton Inference Server
Apache License 2.0
71 stars 36 forks source link

Unable to use FP64 data type in input to xgb model #108

Closed rohitgpt178 closed 3 years ago

rohitgpt178 commented 3 years ago

I am encountering the following error when I try to use FP64 data type for input to an xgboost regression model on fil backend. (Using perf_analyzer to send requests)

original inputs:
[0x0x7f0e9801bd38] input: input__0, type: FP64, original shape: [1,32], batch + shape: [1,32], shape: [32]
override inputs:
inputs:
[0x0x7f0e9801bd38] input: input__0, type: FP64, original shape: [1,32], batch + shape: [1,32], shape: [32]
original requested outputs:
output__0
requested outputs:
output__0

[ERROR] evhtp.c:3101     ODDITY, resuming when not paused?!? :: (errno: None)
double free or corruption (out)
Aborted (core dumped)

Input/output config - 

input [
 {
    name: "input__0"
    data_type: TYPE_FP64
    dims: [ 32 ]
  }
]
output [
 {
    name: "output__0"
    data_type: TYPE_FP32
    dims: [ 1 ]
  }
]

Keeping both input & output FP32, or input FP32 & output FP64 is working fine.

Also I am able to use numpy.float64 input data on the same model using fil in rapidsai container.

The model is an example xgboost regression model generated from here.

wphicks commented 3 years ago

Thanks for reporting this, @rohitgpt178! At present, the FIL backend supports only FP32 input and output, but it is troubling that attempting to use FP64 caused a crash. We'll look into it and update the docs to make it clear that FP32 is required.

In terms of your specific use case, are you concerned about degraded accuracy when using FP32, or are you looking for smoother integration with other models that use FP64 in a pipeline or something else entirely? If the concern is degraded accuracy, FIL internally converts everything to single-precision anyway, so taking advantage of FP64 would require a more significant change, and I doubt you would see a practical difference anyway.

rohitgpt178 commented 3 years ago

Got it @wphicks. We are fine with using FP32 for our use case as far as the accuracy is concerned, though a smoother integration with fp64 might be helpful. I just found that the server is crashing and thought of reporting it. Also I was not aware that FIL converts data to single precision, thanks for informing.

wphicks commented 3 years ago

Yes, thank you very much for reporting the crash! We definitely want to get that fixed, just figuring out what the best solution looks like for you. We'll get to work on it!

wphicks commented 3 years ago

There's now a PR to fix the server crash (#119) and two new issues to explicitly support non-FP32 I/O in the future (#121 and #122). Once we merge #119, we'll close this issue and follow-up on any problems related to I/O type conversion in those new issues.

rohitgpt178 commented 3 years ago

Cool, thanks @wphicks!