search

sql-machine-learning / sqlflow

Brings SQL and AI together.
https://sqlflow.org
Apache License 2.0
5.09k stars 699 forks source link

using sqldataset? #277

Closed yupbank closed 5 years ago

yupbank commented 5 years ago

hey, i think tensorflow provides native sql dataset ops for you to deal with database.

is there any thoughts on that?

FYI https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/data/experimental/ops/readers.py

wangkuiyi commented 5 years ago

I know this SqlDataset from it's document at https://www.tensorflow.org/api_docs/python/tf/data/experimental/SqlDataset. The previous time I checked it's source code I realized it can only read from SQLite. Not sure what's it's current status.

wangkuiyi commented 5 years ago

Just checked the document says the only supported SQL database is still SQLite:

driver_name: A 0-D tf.string tensor containing the database type. Currently, the only supported value is 'sqlite'.

Actually we have colleagues who tried to contribute a dataset operator that can read Alibaba ODPS. He had to add ODPS client library as an additional dependency of TensorFlow. I am not sure how many SQL drivers TensorFlow could take as dependencies. It looks to me reasonable to make these operators external and optional extensions to TensorFlow instead of part of TensorFlow.

yupbank commented 5 years ago

yeah, we have a sigIO from tensorflow here https://github.com/tensorflow/io