MLHandler's typecasting is too restrictive

[jaidevd]

Is something not working as expected? Because MLHandler accepts feature values through URLs, they have to be coerced into the correct types. This can be too restrictive, because the types are inferred from the dataframes that are cached during training. Especially, if a dataframe has a feature which is an integer, MLHandler won't allow it to have a value that is a float.

Steps to reproduce. Please help us reproduce the bug, by sharing:

1. Paste this in a file named xor.csv:

   x,y,z
   0,0,0
   0,1,1
   1,0,1
   1,1,0

2. Use the following gramex config:

   
   url:
       xor:
               pattern: /$YAMLURL/
               handler: MLHandler
               kwargs:
                       data: $YAMLPATH/xor.csv
                       model:
                               class: SVC
                               target_col: z

3. Run gramex, and then try [http://localhost:9988/?x=1.5&y=1.5](http://localhost:9988/?x=1.5&y=1.5), to get the following error

ERROR 26-Feb 20:11:37 web Uncaught exception GET /?x=1.5&y=1.5 (::1) HTTPServerRequest(protocol='http', host='localhost:9988', method='GET', uri='/?x=1.5&y=1.5', version='HTTP/1.1', remote_ip='::1') Traceback (most recent call last): File "/home/jaidevd/anaconda3/lib/python3.7/site-packages/tornado/web.py", line 1592, in _execute result = yield result File "/home/jaidevd/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run value = future.result() File "/home/jaidevd/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 1141, in run yielded = self.gen.throw(exc_info) File "/home/jaidevd/src/gramex/gramex/handlers/mlhandler.py", line 466, in get self._predict, to_predict) File "/home/jaidevd/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run value = future.result() File "/home/jaidevd/anaconda3/lib/python3.7/concurrent/futures/_base.py", line 428, in result return self.get_result() File "/home/jaidevd/anaconda3/lib/python3.7/concurrent/futures/_base.py", line 384, in get_result raise self._exception File "/home/jaidevd/anaconda3/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(self.args, self.kwargs) File "/home/jaidevd/src/gramex/gramex/handlers/mlhandler.py", line 336, in _predict data = self._transform(data, deduplicate=False) File "/home/jaidevd/src/gramex/gramex/handlers/mlhandler.py", line 308, in _transform data[col] = data[col].astype(orgdata[col].dtype) File "/home/jaidevd/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py", line 5882, in astype dtype=dtype, copy=copy, errors=errors, kwargs File "/home/jaidevd/anaconda3/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 581, in astype return self.apply("astype", dtype=dtype, kwargs) File "/home/jaidevd/anaconda3/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 438, in apply applied = getattr(b, f)(kwargs) File "/home/jaidevd/anaconda3/lib/python3.7/site-packages/pandas/core/internals/blocks.py", line 559, in astype return self._astype(dtype, copy=copy, errors=errors, values=values, kwargs) File "/home/jaidevd/anaconda3/lib/python3.7/site-packages/pandas/core/internals/blocks.py", line 643, in _astype values = astype_nansafe(vals1d, dtype, copy=True, kwargs) File "/home/jaidevd/anaconda3/lib/python3.7/site-packages/pandas/core/dtypes/cast.py", line 707, in astype_nansafe return lib.astype_intsafe(arr.ravel(), dtype).reshape(arr.shape) File "pandas/_libs/lib.pyx", line 547, in pandas._libs.lib.astype_intsafe ValueError: invalid literal for int() with base 10: '1.5'

The underlying sklearn model is perfectly capable of classifying x=1.5 and y=1.5.

[pratapvardhan]

Assuming we stick with pandas for reading the csv file. Supporting read_csv(..., dtype={}, ...) should allow users to be more explicit -- on how to infer types?

[jaidevd]

@pratapvardhan We currently don't have a way of letting Gramex users specify dtypes from the yaml or requests. We should, eventually.