[spark-tensorflow-connector] All input data are loaded to infer schema

[manuzhang]

I find all input data are loaded when I just want to print schema with the following code.

spark.read \
    .option("recordType", "Example") \
    .format("tfrecords") \
    .load(path) \
    .printSchema()

[WegenPan]

you can load data like this, and you need to prepare the schema sparkSession.read.format("tfrecords").schema(schema).load(inputPattern)

[manuzhang]

Unless I don't know the schema and want to find out with printSchema

[WegenPan]

Unless I don't know the schema and want to find out with printSchema

I guess the process need to scan the whole data to generate the schema, may be you can try use less data