apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.88k stars 4.27k forks source link

[Feature Request]: Add A Freature to get bigquery schema dynamically from a POJO class. #32311

Open nivedha1 opened 2 months ago

nivedha1 commented 2 months ago

What would you like to happen?

In Bigquery write introduce method "fetchSchemaFromObject" which will dynamically generate schema json from the supplied POJO class. BigQueryIO.write() .to(tableSpec) .fetchSchemaFromObject(Person.class) ..withFormatFunction( (Person elem) -> new TableRow().set("name", elem.name).set("age", elem.age)) .withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED) .withWriteDisposition(WriteDisposition.WRITE_TRUNCATE));

   /** Person.java class **/
   class Person { 
           String name;
            int age;
   }
  /**
      Parse Person Object and construct schema json.
  **/
  fetchSchemaFromObject(Object Person)

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

Ganeshsivakumar commented 1 month ago

hey @nivedha1 could you share more context on why schema needs to be generated here, .fetchSchemaFromObject(Person.class) when Person object is directly used.