Closed tsai0009 closed 4 years ago
Hello,
I have been testing the Kite-sdk maven dependency due to a recommendation made by this question:
https://stackoverflow.com/questions/46556614/is-there-a-way-to-programmatically-convert-json-to-avro-schema/46566592#46566592
The problem is, when I use the small script I wrote:
package avro_schema_builder; import java.io.File; import java.io.FileReader; import org.json.simple.JSONArray; import org.json.simple.parser.JSONParser; import org.kitesdk.data.spi.JsonUtil; public class avro_schema_builder { public static void main(String[] args) throws Exception { File file = new File("src/avro_schema_builder/avro.json"); String path = file.getCanonicalPath(); JSONParser jsonParser = new JSONParser(); FileReader reader = new FileReader(path); Object obj = jsonParser.parse(reader); JSONArray json = (JSONArray) obj; String avroSchema = JsonUtil.inferSchema(JsonUtil.parse(json.toString()), "test").toString(); System.out.println(avroSchema); } }
And a random online JSON I found:
[ { "webapp": { "asd": { "ase": { "asf": 0 } } }, "servlet": [ { "servletname": "cofaxCDS", "servletclass": "org.cofax.cds.CDSServlet", "initparam": { "configGlossary_installationAt": "Philadelphia, PA", "configGlossary_adminEmail": "ksm@pobox.com", "configGlossary_poweredBy": "Cofax", "configGlossary_poweredByIcon": "/images/cofax.gif", "configGlossary_staticPath": "/content/static", "templateProcessorClass": "org.cofax.WysiwygTemplate", "templateLoaderClass": "org.cofax.FilesTemplateLoader", "templatePath": "templates", "templateOverridePath": "", "defaultListTemplate": "listTemplate.htm", "defaultFileTemplate": "articleTemplate.htm", "useJSP": false, "jspListTemplate": "listTemplate.jsp", "jspFileTemplate": "articleTemplate.jsp", "cachePackageTagsTrack": 200, "cachePackageTagsStore": 200, "cachePackageTagsRefresh": 60, "cacheTemplatesTrack": 100, "cacheTemplatesStore": 50, "cacheTemplatesRefresh": 15, "cachePagesTrack": 200, "cachePagesStore": 100, "cachePagesRefresh": 10, "cachePagesDirtyRead": 10, "searchEngineListTemplate": "forSearchEnginesList.htm", "searchEngineFileTemplate": "forSearchEngines.htm", "searchEngineRobotsDb": "WEBINF/robots.db", "useDataStore": true, "dataStoreClass": "org.cofax.SqlDataStore", "redirectionClass": "org.cofax.SqlRedirection", "dataStoreName": "cofax", "dataStoreDriver": "com.microsoft.jdbc.sqlserver.SQLServerDriver", "dataStoreUrl": "jdbc:microsoft:sqlserver://LOCALHOST:1433;DatabaseName=goon", "dataStoreUser": "sa", "dataStorePassword": "dataStoreTestQuery", "dataStoreTestQuery": "SET NOCOUNT ON;select test='test';", "dataStoreLogFile": "/usr/local/tomcat/logs/datastore.log", "dataStoreInitConns": 10, "dataStoreMaxConns": 100, "dataStoreConnUsageLimit": 100, "dataStoreLogLevel": "debug", "maxUrlLength": 500 } }, { "servletname": "cofaxEmail", "servletclass": "org.cofax.cds.EmailServlet", "initparam": { "mailHost": "mail1", "mailHostOverride": "mail2" } }, { "servletclass": "org.cofax.cds.AdminServlet" }, { "servletname": "fileServlet", "servletclass": "org.cofax.cds.FileServlet" }, { "servletname": "cofaxTools", "servletclass": "org.cofax.cms.CofaxToolsServlet", "initparam": { "templatePath": "toolstemplates/", "log": 1, "logLocation": "/usr/local/tomcat/logs/CofaxTools.log", "logMaxSize": "", "dataLog": 1, "dataLogLocation": "/usr/local/tomcat/logs/dataLog.log", "dataLogMaxSize": "", "removePageCache": "/content/admin/remove?cache=pages&id=", "removeTemplateCache": "/content/admin/remove?cache=templates&id=", "fileTransferFolder": "/usr/local/tomcat/webapps/content/fileTransferFolder", "lookInContext": 1, "adminGroupID": 4, "betaServer": true } } ], "servletmapping": { "cofaxCDS": "/", "cofaxEmail": "/cofaxutil/aemail/*", "cofaxAdmin": "/admin/*", "fileServlet": "/static/*", "cofaxTools": "/tools/*" }, "taglib": { "tagliburi": "cofax.tld", "tagliblocation": "/WEBINF/tlds/cofax.tld" } } ]
I get the wrong output:
{ "type": "array", "items": { "type": "record", "name": "test", "fields": [{ "name": "webapp", "type": { "type": "record", "name": "webapp", "fields": [{ "name": "asd", "type": { "type": "record", "name": "webapp", "namespace": "asd", "fields": [{ "name": "ase", "type": { "type": "record", "name": "webapp", "namespace": "ase.asd", "fields": [{ "name": "asf", "type": "int", "doc": "Type inferred from '0'" } ] }, "doc": "Type inferred from '{\"asf\":0}'" } ] }, "doc": "Type inferred from '{\"ase\":{\"asf\":0}}'" } ] }, "doc": "Type inferred from '{\"asd\":{\"ase\":{\"asf\":0}}}'" }, { "name": "servletmapping", "type": { "type": "record", "name": "servletmapping", "fields": [{ "name": "cofaxAdmin", "type": "string", "doc": "Type inferred from '\"/admin/*\"'" }, { "name": "cofaxCDS", "type": "string", "doc": "Type inferred from '\"/\"'" }, { "name": "cofaxEmail", "type": "string", "doc": "Type inferred from '\"/cofaxutil/aemail/*\"'" }, { "name": "fileServlet", "type": "string", "doc": "Type inferred from '\"/static/*\"'" }, { "name": "cofaxTools", "type": "string", "doc": "Type inferred from '\"/tools/*\"'" } ] }, "doc": "Type inferred from '{\"cofaxAdmin\":\"/admin/*\",\"cofaxCDS\":\"/\",\"cofaxEmail\":\"/cofaxutil/aemail/*\",\"fileServlet\":\"/static/*\",\"cofaxTools\":\"/tools/*\"}'" }, { "name": "taglib", "type": { "type": "record", "name": "taglib", "fields": [{ "name": "tagliburi", "type": "string", "doc": "Type inferred from '\"cofax.tld\"'" }, { "name": "tagliblocation", "type": "string", "doc": "Type inferred from '\"/WEBINF/tlds/cofax.tld\"'" } ] }, "doc": "Type inferred from '{\"tagliburi\":\"cofax.tld\",\"tagliblocation\":\"/WEBINF/tlds/cofax.tld\"}'" }, { "name": "servlet", "type": { "type": "array", "items": { "type": "record", "name": "servlet", "fields": [{ "name": "servletclass", "type": "string", "doc": "Type inferred from '\"org.cofax.cds.CDSServlet\"'" }, { "name": "initparam", "type": ["null", { "type": "record", "name": "servlet", "namespace": "initparam", "fields": [{ "name": "cachePackageTagsTrack", "type": ["null", "int"], "doc": "Type inferred from '200'", "default": null }, { "name": "redirectionClass", "type": ["null", "string"], "doc": "Type inferred from '\"org.cofax.SqlRedirection\"'", "default": null }, { "name": "jspFileTemplate", "type": ["null", "string"], "doc": "Type inferred from '\"articleTemplate.jsp\"'", "default": null }, { "name": "cacheTemplatesRefresh", "type": ["null", "int"], "doc": "Type inferred from '15'", "default": null }, { "name": "dataStorePassword", "type": ["null", "string"], "doc": "Type inferred from '\"dataStoreTestQuery\"'", "default": null }, { "name": "dataStoreClass", "type": ["null", "string"], "doc": "Type inferred from '\"org.cofax.SqlDataStore\"'", "default": null }, { "name": "cacheTemplatesTrack", "type": ["null", "int"], "doc": "Type inferred from '100'", "default": null }, { "name": "configGlossary_poweredByIcon", "type": ["null", "string"], "doc": "Type inferred from '\"/images/cofax.gif\"'", "default": null }, { "name": "searchEngineFileTemplate", "type": ["null", "string"], "doc": "Type inferred from '\"forSearchEngines.htm\"'", "default": null }, { "name": "configGlossary_adminEmail", "type": ["null", "string"], "doc": "Type inferred from '\"ksm@pobox.com\"'", "default": null }, { "name": "defaultFileTemplate", "type": ["null", "string"], "doc": "Type inferred from '\"articleTemplate.htm\"'", "default": null }, { "name": "templateProcessorClass", "type": ["null", "string"], "doc": "Type inferred from '\"org.cofax.WysiwygTemplate\"'", "default": null }, { "name": "configGlossary_installationAt", "type": ["null", "string"], "doc": "Type inferred from '\"Philadelphia, PA\"'", "default": null }, { "name": "searchEngineListTemplate", "type": ["null", "string"], "doc": "Type inferred from '\"forSearchEnginesList.htm\"'", "default": null }, { "name": "cachePagesStore", "type": ["null", "int"], "doc": "Type inferred from '100'", "default": null }, { "name": "useDataStore", "type": ["null", "boolean"], "doc": "Type inferred from 'true'", "default": null }, { "name": "configGlossary_poweredBy", "type": ["null", "string"], "doc": "Type inferred from '\"Cofax\"'", "default": null }, { "name": "templateLoaderClass", "type": ["null", "string"], "doc": "Type inferred from '\"org.cofax.FilesTemplateLoader\"'", "default": null }, { "name": "cachePagesTrack", "type": ["null", "int"], "doc": "Type inferred from '200'", "default": null }, { "name": "searchEngineRobotsDb", "type": ["null", "string"], "doc": "Type inferred from '\"WEBINF/robots.db\"'", "default": null }, { "name": "cachePagesDirtyRead", "type": ["null", "int"], "doc": "Type inferred from '10'", "default": null }, { "name": "cachePackageTagsStore", "type": ["null", "int"], "doc": "Type inferred from '200'", "default": null }, { "name": "cachePackageTagsRefresh", "type": ["null", "int"], "doc": "Type inferred from '60'", "default": null }, { "name": "configGlossary_staticPath", "type": ["null", "string"], "doc": "Type inferred from '\"/content/static\"'", "default": null }, { "name": "dataStoreConnUsageLimit", "type": ["null", "int"], "doc": "Type inferred from '100'", "default": null }, { "name": "useJSP", "type": ["null", "boolean"], "doc": "Type inferred from 'false'", "default": null }, { "name": "dataStoreLogLevel", "type": ["null", "string"], "doc": "Type inferred from '\"debug\"'", "default": null }, { "name": "dataStoreUrl", "type": ["null", "string"], "doc": "Type inferred from '\"jdbc:microsoft:sqlserver://LOCALHOST:1433;DatabaseName=goon\"'", "default": null }, { "name": "templatePath", "type": ["null", "string"], "doc": "Type inferred from '\"templates\"'", "default": null }, { "name": "cacheTemplatesStore", "type": ["null", "int"], "doc": "Type inferred from '50'", "default": null }, { "name": "jspListTemplate", "type": ["null", "string"], "doc": "Type inferred from '\"listTemplate.jsp\"'", "default": null }, { "name": "dataStoreTestQuery", "type": ["null", "string"], "doc": "Type inferred from '\"SET NOCOUNT ON;select test='test';\"'", "default": null }, { "name": "dataStoreMaxConns", "type": ["null", "int"], "doc": "Type inferred from '100'", "default": null }, { "name": "dataStoreName", "type": ["null", "string"], "doc": "Type inferred from '\"cofax\"'", "default": null }, { "name": "maxUrlLength", "type": ["null", "int"], "doc": "Type inferred from '500'", "default": null }, { "name": "templateOverridePath", "type": ["null", "string"], "doc": "Type inferred from '\"\"'", "default": null }, { "name": "cachePagesRefresh", "type": ["null", "int"], "doc": "Type inferred from '10'", "default": null }, { "name": "dataStoreDriver", "type": ["null", "string"], "doc": "Type inferred from '\"com.microsoft.jdbc.sqlserver.SQLServerDriver\"'", "default": null }, { "name": "dataStoreUser", "type": ["null", "string"], "doc": "Type inferred from '\"sa\"'", "default": null }, { "name": "dataStoreLogFile", "type": ["null", "string"], "doc": "Type inferred from '\"/usr/local/tomcat/logs/datastore.log\"'", "default": null }, { "name": "defaultListTemplate", "type": ["null", "string"], "doc": "Type inferred from '\"listTemplate.htm\"'", "default": null }, { "name": "dataStoreInitConns", "type": ["null", "int"], "doc": "Type inferred from '10'", "default": null }, { "name": "mailHost", "type": ["null", "string"], "doc": "Type inferred from '\"mail1\"'", "default": null }, { "name": "mailHostOverride", "type": ["null", "string"], "doc": "Type inferred from '\"mail2\"'", "default": null }, { "name": "dataLogMaxSize", "type": ["null", "string"], "doc": "Type inferred from '\"\"'", "default": null }, { "name": "log", "type": ["null", "int"], "doc": "Type inferred from '1'", "default": null }, { "name": "logMaxSize", "type": ["null", "string"], "doc": "Type inferred from '\"\"'", "default": null }, { "name": "fileTransferFolder", "type": ["null", "string"], "doc": "Type inferred from '\"/usr/local/tomcat/webapps/content/fileTransferFolder\"'", "default": null }, { "name": "removeTemplateCache", "type": ["null", "string"], "doc": "Type inferred from '\"/content/admin/remove?cache=templates&id=\"'", "default": null }, { "name": "dataLogLocation", "type": ["null", "string"], "doc": "Type inferred from '\"/usr/local/tomcat/logs/dataLog.log\"'", "default": null }, { "name": "lookInContext", "type": ["null", "int"], "doc": "Type inferred from '1'", "default": null }, { "name": "removePageCache", "type": ["null", "string"], "doc": "Type inferred from '\"/content/admin/remove?cache=pages&id=\"'", "default": null }, { "name": "adminGroupID", "type": ["null", "int"], "doc": "Type inferred from '4'", "default": null }, { "name": "betaServer", "type": ["null", "boolean"], "doc": "Type inferred from 'true'", "default": null }, { "name": "logLocation", "type": ["null", "string"], "doc": "Type inferred from '\"/usr/local/tomcat/logs/CofaxTools.log\"'", "default": null }, { "name": "dataLog", "type": ["null", "int"], "doc": "Type inferred from '1'", "default": null } ] } ], "doc": "Type inferred from '{\"cachePackageTagsTrack\":200,\"redirectionClass\":\"org.cofax.SqlRedirection\",\"jspFileTemplate\":\"articleTemplate.jsp\",\"cacheTemplatesRefresh\":15,\"dataStorePassword\":\"dataStoreTestQuery\",\"dataStoreClass\":\"org.cofax.SqlDataStore\",\"cacheTemplatesTrack\":100,\"configGlossary_poweredByIcon\":\"/images/cofax.gif\",\"searchEngineFileTemplate\":\"forSearchEngines.htm\",\"configGlossary_adminEmail\":\"ksm@pobox.com\",\"defaultFileTemplate\":\"articleTemplate.htm\",\"templateProcessorClass\":\"org.cofax.WysiwygTemplate\",\"configGlossary_installationAt\":\"Philadelphia, PA\",\"searchEngineListTemplate\":\"forSearchEnginesList.htm\",\"cachePagesStore\":100,\"useDataStore\":true,\"configGlossary_poweredBy\":\"Cofax\",\"templateLoaderClass\":\"org.cofax.FilesTemplateLoader\",\"cachePagesTrack\":200,\"searchEngineRobotsDb\":\"WEBINF/robots.db\",\"cachePagesDirtyRead\":10,\"cachePackageTagsStore\":200,\"cachePackageTagsRefresh\":60,\"configGlossary_staticPath\":\"/content/static\",\"dataStoreConnUsageLimit\":100,\"useJSP\":false,\"dataStoreLogLevel\":\"debug\",\"dataStoreUrl\":\"jdbc:microsoft:sqlserver://LOCALHOST:1433;DatabaseName=goon\",\"templatePath\":\"templates\",\"cacheTemplatesStore\":50,\"jspListTemplate\":\"listTemplate.jsp\",\"dataStoreTestQuery\":\"SET NOCOUNT ON;select test='test';\",\"dataStoreMaxConns\":100,\"dataStoreName\":\"cofax\",\"maxUrlLength\":500,\"templateOverridePath\":\"\",\"cachePagesRefresh\":10,\"dataStoreDriver\":\"com.microsoft.jdbc.sqlserver.SQLServerDriver\",\"dataStoreUser\":\"sa\",\"dataStoreLogFile\":\"/usr/local/tomcat/logs/datastore.log\",\"defaultListTemplate\":\"listTemplate.htm\",\"dataStoreInitConns\":10}'", "default": null }, { "name": "servletname", "type": ["null", "string"], "doc": "Type inferred from '\"cofaxCDS\"'", "default": null } ] } }, "doc": "Type inferred from '[{\"servletclass\":\"org.cofax.cds.CDSServlet\",\"initparam\":{\"cachePackageTagsTrack\":200,\"redirectionClass\":\"org.cofax.SqlRedirection\",\"jspFileTemplate\":\"articleTemplate.jsp\",\"cacheTemplatesRefresh\":15,\"dataStorePassword\":\"dataStoreTestQuery\",\"dataStoreClass\":\"org.cofax.SqlDataStore\",\"cacheTemplatesTrack\":100,\"configGlossary_poweredByIcon\":\"/images/cofax.gif\",\"searchEngineFileTemplate\":\"forSearchEngines.htm\",\"configGlossary_adminEmail\":\"ksm@pobox.com\",\"defaultFileTemplate\":\"articleTemplate.htm\",\"templateProcessorClass\":\"org.cofax.WysiwygTemplate\",\"configGlossary_installationAt\":\"Philadelphia, PA\",\"searchEngineListTemplate\":\"forSearchEnginesList.htm\",\"cachePagesStore\":100,\"useDataStore\":true,\"configGlossary_poweredBy\":\"Cofax\",\"templateLoaderClass\":\"org.cofax.FilesTemplateLoader\",\"cachePagesTrack\":200,\"searchEngineRobotsDb\":\"WEBINF/robots.db\",\"cachePagesDirtyRead\":10,\"cachePackageTagsStore\":200,\"cachePackageTagsRefresh\":60,\"configGlossary_staticPath\":\"/content/static\",\"dataStoreConnUsageLimit\":100,\"useJSP\":false,\"dataStoreLogLevel\":\"debug\",\"dataStoreUrl\":\"jdbc:microsoft:sqlserver://LOCALHOST:1433;DatabaseName=goon\",\"templatePath\":\"templates\",\"cacheTemplatesStore\":50,\"jspListTemplate\":\"listTemplate.jsp\",\"dataStoreTestQuery\":\"SET NOCOUNT ON;select test='test';\",\"dataStoreMaxConns\":100,\"dataStoreName\":\"cofax\",\"maxUrlLength\":500,\"templateOverridePath\":\"\",\"cachePagesRefresh\":10,\"dataStoreDriver\":\"com.microsoft.jdbc.sqlserver.SQLServerDriver\",\"dataStoreUser\":\"sa\",\"dataStoreLogFile\":\"/usr/local/tomcat/logs/datastore.log\",\"defaultListTemplate\":\"listTemplate.htm\",\"dataStoreInitConns\":10},\"servletname\":\"cofaxCDS\"},{\"servletclass\":\"org.cofax.cds.EmailServlet\",\"initparam\":{\"mailHost\":\"mail1\",\"mailHostOverride\":\"mail2\"},\"servletname\":\"cofaxEmail\"},{\"servletclass\":\"org.cofax.cds.AdminServlet\"},{\"servletclass\":\"org.cofax.cds.FileServlet\",\"servletname\":\"fileServlet\"},{\"servletclass\":\"org.cofax.cms.CofaxToolsServlet\",\"initparam\":{\"dataLogMaxSize\":\"\",\"log\":1,\"logMaxSize\":\"\",\"templatePath\":\"toolstemplates/\",\"fileTransferFolder\":\"/usr/local/tomcat/webapps/content/fileTransferFolder\",\"removeTemplateCache\":\"/content/admin/remove?cache=templates&id=\",\"dataLogLocation\":\"/usr/local/tomcat/logs/dataLog.log\",\"lookInContext\":1,\"removePageCache\":\"/content/admin/remove?cache=pages&id=\",\"adminGroupID\":4,\"betaServer\":true,\"logLocation\":\"/usr/local/tomcat/logs/CofaxTools.log\",\"dataLog\":1},\"servletname\":\"cofaxTools\"}]'" } ] } }
In the first segment, webapp should not be repeated more than twice. It should be:
name: webapp, type: { name: webapp, type: record, fields: [ {name: asd, type: {type: 'record, name: 'asd}]}
not
name: webapp, namespace: asd
PS, is there a way to get rid of the "doc"s and also avoid the dot notation?
Tried with validator python code and it passes, still have my doubts about the format.
Hello,
I have been testing the Kite-sdk maven dependency due to a recommendation made by this question:
https://stackoverflow.com/questions/46556614/is-there-a-way-to-programmatically-convert-json-to-avro-schema/46566592#46566592
The problem is, when I use the small script I wrote:
And a random online JSON I found:
I get the wrong output:
In the first segment, webapp should not be repeated more than twice. It should be:
not
PS, is there a way to get rid of the "doc"s and also avoid the dot notation?