ravro unable to read acro file for my schema

[loicmathieu]

Hello,

I successfully read an avro file from this exemple : https://github.com/miguno/avro-cli-examples/blob/master/twitter.avro but when I try with my schema, a more complex one, ravro gives the following error :

Error in mapply(parse_avro, x_df, schema$fields, MoreArgs = list(flatten = flatten,  :
  zero-length inputs cannot be mixed with those of non-zero length

Here is the schema that I use to create my avro file

{
    "namespace":"com.kiabi.avro.web.orders",
    "type":"record",
    "doc":"This Schema describes about web orders ",
    "name":"Commande",
    "fields": [
        {"name": "metadata",
            "type": {
                "name": "Metadata", "type": "record",
                "fields": [
                    {"name": "date_evenement", "type" : "long", "description": "date d'integration dans la zone de staging - timestamp UNIX"},
                    {"name": "type_evenement", "type" : "string", "description": "Creation, Modication, Suppression"},
                    {"name": "date_integration", "type" : ["null","long"], "description": "date d'integration dans la zone de data - timestamp UNIX"}
                ]
            }
        },
        {"name": "entete_commande",
            "type": {
                "name": "EnteteCommande", "type": "record",
                "fields": [
                    {"name": "code_site_web", "type" : "string"},
                    {"name": "code_cde_web", "type" : "string"},
                    {"name": "no_cli", "type" : "long"},
                    {"name": "no_adresse_cli", "type" : ["null","long"]},
                    {"name": "no_adresse_fact", "type" : ["null","long"]},
                    {"name": "code_etat_cde", "type" : ["null","string"]},
                    {"name": "date_etat_cde", "type" : ["null","long"]},
                    {"name": "code_langue", "type" : ["null","string"]},
                    {"name": "flag_remise_fid", "type" : ["null","string"]},
                    {"name": "flag_dem_carte_fid", "type" : ["null","string"]},
                    {"name": "flag_ach_mag", "type" : ["null","string"]},
                    {"name": "code_orig_connexion", "type" : ["null","long"]},
                    {"name": "no_carte_fid", "type" : ["null","string"]},
                    {"name": "lib_msg_cadeau", "type" : ["null","string"]},
                    {"name": "code_dev", "type" : ["null","string"]},
                    {"name": "code_mode_pay", "type" : ["null","string"]},
                    {"name": "flag_pay_val", "type" : ["null","string"]},
                    {"name": "flag_pay_ann", "type" : ["null","string"]},
                    {"name": "date_pay_ann", "type" : ["null","long"]},
                    {"name": "no_colis_total", "type" : ["null","long"]},
                    {"name": "code_orig_affectation_ca", "type" : ["null","string"]},
                    {"name": "code_etab_affectation_ca", "type" : ["null","string"]},
                    {"name": "code_postal_affectation_ca", "type" : ["null","string"]},
                    {"name": "code_pays_affectation_ca", "type" : ["null","string"]},
                    {"name": "flag_affectation_ca", "type" : ["null","string"]},
                    {"name": "histo_etat_commande",
                        "doc": "on a 1 a N historique de l'etat de la commande",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "HistoEtatCommande",
                                "fields":[
                                    {"name": "id_histo_etat_cde", "type" : "long"},
                                    {"name": "no_expe", "type" : ["null","string"]},
                                    {"name": "no_reliquat", "type" : ["null","string"]},
                                    {"name": "code_etat_cde", "type" : "string"},
                                    {"name": "date_etat_cde", "type" : "long"}
                                ]
                            }
                        }
                    },
                    {"name": "details_commande",
                        "doc": "on a 1 a N detail par commande",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "DetailsCommande",
                                "fields":[
                                    {"name": "no_ligne", "type" : "long"},
                                    {"name": "no_expe", "type" : "string"},
                                    {"name": "no_reliquat", "type" : "string"},
                                    {"name": "code_produit", "type" : "string"},
                                    {"name": "code_type_produit", "type" : "string"},
                                    {"name": "code_orig_cat", "type" : "string"},
                                    {"name": "code_produit_reel", "type" : ["null","string"]},
                                    {"name": "code_colis", "type" : ["null","string"]},
                                    {"name": "code_etat_lig_cde", "type" : ["null","string"]},
                                    {"name": "date_etat_lig_cde", "type" : ["null","long"]},
                                    {"name": "code_etab_log", "type" : ["null","string"]},
                                    {"name": "qte_cde", "type" : ["null","long"]},
                                    {"name": "qte_ano_stock", "type" : ["null","long"]},
                                    {"name": "qte_dem_prel", "type" : ["null","long"]},
                                    {"name": "qte_prel", "type" : ["null","long"]},
                                    {"name": "qte_expe_cli", "type" : ["null","long"]},
                                    {"name": "no_cde_cli", "type" : ["null","long"]},
                                    {"name": "flag_picking", "type" : ["null","string"]},
                                    {"name": "flag_envoi_site", "type" : ["null","string"]},
                                    {"name": "pud_pvttc_caisse", "type" : ["null","long"]},
                                    {"name": "valod_pud_pvttc_caisse", "type" : ["null","long"]},
                                    {"name": "code_operation", "type" : ["null","string"]},
                                    {"name": "taux_tva", "type" : ["null","double"]},
                                    {"name": "pud_remb", "type" : ["null","long"]},
                                    {"name": "valod_pud_remb", "type" : ["null","long"]},
                                    {"name": "code_pays_fab", "type" : ["null","string"]},
                                    {"name": "pud_remb_orig", "type" : ["null","long"]},
                                    {"name": "valod_pud_remb_orig", "type" : ["null","long"]},
                                    {"name": "pud_pvttc_caisse_orig", "type" : ["null","long"]},
                                    {"name": "valod_pud_pvttc_caisse_orig", "type" : ["null","long"]},
                                    {"name": "taux_tva_orig", "type" : ["null","double"]},
                                    {"name": "no_distro", "type" : ["null","string"]},
                                    {"name": "no_prio_expe", "type" : ["null","long"]},
                                    {"name": "qte_rec", "type" : ["null","long"]},
                                    {"name": "qte_ano_rec", "type" : ["null","long"]},
                                    {"name": "qte_ano_tri", "type" : ["null","long"]},
                                    {"name": "pud_pvttc_affichage_web", "type" : ["null","long"]},
                                    {"name": "valod_pud_pvttc_affichage_web", "type" : ["null","long"]},                                    
                                    {"name": "pud_pvttc_affichage_web_orig", "type" : ["null","long"]},
                                    {"name": "valod_pud_pvttc_affichage_web_orig", "type" : ["null","long"]},
                                    {"name": "qte_distro_expe", "type" : ["null","long"]},
                                    {"name": "type_tri", "type" : ["null","string"]},
                                    {"name": "code_etab_tri", "type" : ["null","string"]},
                                    {"name": "histo_details_commande",
                                        "doc": "on a 1 a N histo par detail",
                                        "type": {
                                            "type": "array",
                                            "items": {
                                                "type": "record",
                                                "name": "HistoDetailsCommande",
                                                "fields":[
                                                    {"name": "code_etat_lig_cde", "type" : "string"},
                                                    {"name": "date_etat_lig_cde", "type" : "long"},
                                                    {"name": "id_cmt", "type" : ["null","long"]}
                                                ]
                                            }
                                        }
                                    }
                                ]
                            }
                        }
                    }, 
                    {"name": "financement_commande",
                        "doc": "on a 1 a N financement par commande",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "FinancementCommande",
                                "fields":[
                                    {"name": "no_expe", "type" : "string"},
                                    {"name": "type_financ", "type" : "string"},
                                    {"name": "mode_financ", "type" : "string"},
                                    {"name": "code_dev_financ", "type" : "string"},
                                    {"name": "no_ret", "type" : "long"},
                                    {"name": "mnt_financ", "type" : ["null","long"]},
                                    {"name": "valod_mnt_financ", "type" : ["null","long"]},
                                    {"name": "flag_envoi_site", "type" : ["null","string"]},
                                    {"name": "date_envoi_site", "type" : ["null","long"]},
                                    {"name": "flag_envoi_livcai", "type" : ["null","string"]},
                                    {"name": "date_envoi_livcai", "type" : ["null","long"]},
                                    {"name": "flag_val_site", "type" : ["null","string"]},
                                    {"name": "date_val", "type" : ["null","long"]},
                                    {"name": "mnt_pay_liv", "type" : ["null","long"]},
                                    {"name": "valod_mnt_pay_liv", "type" : ["null","long"]},
                                    {"name": "flag_financ_partiel", "type" : ["null","string"]},
                                    {"name": "flag_transaction_recu", "type" : ["null","string"]},
                                    {"name": "id_transaction", "type" : ["null","long"]},
                                    {"name": "flag_histo_livcai", "type" : ["null","string"]}
                                ]
                            }
                        }
                    },
                    {"name": "bon_commande",
                        "doc": "on a 1 a N bon utilise par commande",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "BonCommande",
                                "fields":[
                                    {"name": "no_bon", "type" : "string"},
                                    {"name": "code_cli_site_web", "type" : ["null","string"]},
                                    {"name": "mnt_financ_bon", "type" : "long"},
                                    {"name": "valod_mnt_financ_bon", "type" : ["null","long"]},
                                    {"name": "code_dev_bon", "type" : "string"},
                                    {"name": "date_blocage_bon", "type" : ["null","long"]},
                                    {"name": "motif_blocage_bon", "type" : ["null","string"]}
                                ]
                            }
                        }
                    },
                    {"name": "bon_emis_commande",
                        "doc": "on a 1 a N bon emis par commande (suite retour)",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "BonEmisCommande",
                                "fields":[
                                    {"name": "no_bon", "type" : "string"},
                                    {"name": "no_expe", "type" : "string"},
                                    {"name": "no_ret", "type" : "long"},
                                    {"name": "val_bon", "type" : "long"},
                                    {"name": "valod_val_bon", "type" : ["null","long"]},
                                    {"name": "date_appli", "type" : "long"},
                                    {"name": "code_dev_bon", "type" : "string"},
                                    {"name": "code_cli_site_web", "type" : ["null","string"]}
                                ]
                            }
                        }
                    },
                    {"name": "expe_commande",
                        "doc": "on a 1 a N expedition par commande",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "ExpeCommande",
                                "fields":[
                                    {"name": "no_expe", "type" : "string"},
                                    {"name": "no_reliquat", "type" : "string"},
                                    {"name": "code_etat_expe", "type" : "string"},
                                    {"name": "date_etat_expe", "type" : "long"},
                                    {"name": "no_cli", "type" : "long"},
                                    {"name": "no_adresse_liv", "type" : ["null","long"]},
                                    {"name": "code_type_liv", "type" : "string"},
                                    {"name": "flag_mono_cv", "type" : "string"},
                                    {"name": "flag_mono_entrepot", "type" : "string"},
                                    {"name": "no_pack", "type" : ["null","string"]},
                                    {"name": "code_relais", "type" : ["null","string"]},
                                    {"name": "code_pays", "type" : ["null","string"]},
                                    {"name": "code_transport", "type" : ["null","string"]},
                                    {"name": "mnt_frais_port", "type" : ["null","long"]},
                                    {"name": "valod_mnt_frais_port", "type" : ["null","long"]},
                                    {"name": "flag_envoi_ca", "type" : ["null","string"]},
                                    {"name": "flag_prel_cde", "type" : ["null","string"]},
                                    {"name": "flag_export", "type" : ["null","string"]},
                                    {"name": "code_transport_reel", "type" : ["null","string"]},
                                    {"name": "code_type_liv_reel", "type" : ["null","string"]}
                                ]
                            }
                        }
                    },
                    {"name": "retour_commande",
                        "doc": "on a 1 a N retour par commande",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "RetourCommande",
                                "fields":[
                                    {"name": "no_expe", "type" : "string"},
                                    {"name": "no_reliquat", "type" : "string"},
                                    {"name": "no_ret", "type" : "long"},
                                    {"name": "date_saisie_ret", "type" : ["null","long"]},
                                    {"name": "flag_remb_bloque", "type" : ["null","string"]},
                                    {"name": "date_remb_deblocage", "type" : ["null","long"]},
                                    {"name": "code_transport_ret", "type" : ["null","string"]},
                                    {"name": "code_user_creation", "type" : ["null","string"]},
                                    {"name": "date_user_maj", "type" : ["null","long"]},
                                    {"name": "code_user_maj", "type" : ["null","string"]},
                                    {"name": "flag_envoi_ca", "type" : ["null","string"]},
                                    {"name": "date_envoi_ca", "type" : ["null","long"]},
                                    {"name": "cmt_ret", "type" : ["null","string"]},
                                    {"name": "flag_histo_ca", "type" : ["null","string"]},
                                    {"name": "details_retour_commande",
                                        "doc": "on a 1 a N detail pour chaque retour de commande",
                                        "type": {
                                            "type": "array",
                                            "items": {
                                                "type": "record",
                                                "name": "DetailsRetourCommande",
                                                "fields":[
                                                    {"name": "no_expe", "type" : "string"},
                                                    {"name": "no_reliquat", "type" : "string"},
                                                    {"name": "no_ligne", "type" : "long"},
                                                    {"name": "code_produit_reel", "type" : ["null","string"]},
                                                    {"name": "qte_ret", "type" : ["null","long"]},
                                                    {"name": "code_motif_ret", "type" : ["null","string"]},
                                                    {"name": "code_etab_emet", "type" : ["null","string"]},
                                                    {"name": "code_etab_dest", "type" : ["null","string"]},
                                                    {"name": "mnt_remb", "type" : ["null","long"]},
                                                    {"name": "valod_mnt_remb", "type" : ["null","long"]},
                                                    {"name": "code_motif_remb", "type" : ["null","string"]},
                                                    {"name": "code_mode_pay_remb", "type" : ["null","string"]},
                                                    {"name": "date_remb_dem", "type" : ["null","long"]},
                                                    {"name": "flag_remb_val", "type" : ["null","string"]},
                                                    {"name": "date_remb_val", "type" : ["null","long"]},
                                                    {"name": "flag_ces", "type" : ["null","string"]},
                                                    {"name": "flag_ces_val", "type" : ["null","string"]},
                                                    {"name": "date_extr_ces", "type" : ["null","long"]},
                                                    {"name": "no_colis_ces", "type" : ["null","long"]}
                                                ]
                                            }
                                        }
                                    }
                                ]
                            }
                        }
                    },
                    {"name": "colis",
                        "doc": "on a 1 a N colis par commande",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "ColisCommande",
                                "fields":[
                                    {"name": "no_expe", "type" : "string"},
                                    {"name": "no_reliquat", "type" : "string"},
                                    {"name": "code_colis", "type" : "string"},
                                    {"name": "code_etat_colis", "type" : "string"},
                                    {"name": "date_etat_colis", "type" : ["null","long"]},
                                    {"name": "poids_colis", "type" : ["null","double"]},
                                    {"name": "code_transport", "type" : "string"},
                                    {"name": "code_colis_prec", "type" : ["null","string"]},
                                    {"name": "code_colis_suiv", "type" : ["null","string"]},
                                    {"name": "no_bord", "type" : ["null","string"]},
                                    {"name": "code_barre_colis", "type" : ["null","string"]},
                                    {"name": "date_envoi_edi", "type" : ["null","long"]},
                                    {"name": "no_ref_transaction_pay", "type" : ["null","string"]},
                                    {"name": "type_ret", "type" : ["null","string"]},
                                    {"name": "date_ret", "type" : ["null","long"]},
                                    {"name": "no_colis", "type" : ["null","long"]},
                                    {"name": "no_fact", "type" : ["null","string"]},
                                    {"name": "code_type_liv_reel", "type" : ["null","string"]},
                                    {"name": "mnt_pay_liv", "type" : ["null","long"]},
                                    {"name": "valod_mnt_pay_liv", "type" : ["null","long"]},
                                    {"name": "date_retrait_max", "type" : ["null","long"]},
                                    {"name": "nom_cli_retrait", "type" : ["null","string"]},
                                    {"name": "code_relais_reel", "type" : ["null","string"]},
                                    {"name": "code_colis_secondaire", "type" : ["null","string"]},
                                    {"name": "type_emballage", "type" : ["null","string"]},
                                    {"name": "financement_colis",
                                        "doc": "on a 1 a N financement par colis",
                                        "type": {
                                            "type": "array",
                                            "items": {
                                                "type": "record",
                                                "name": "FinancementColis",
                                                "fields":[
                                                    {"name": "type_financ", "type" : "string"},
                                                    {"name": "mode_financ", "type" : "string"},
                                                    {"name": "code_dev_financ", "type" : ["null","string"]},
                                                    {"name": "mnt_financ", "type" : ["null","double"]},
                                                    {"name": "valod_mnt_financ", "type" : ["null","double"]},
                                                    {"name": "trt_creation_enr", "type" : ["null","string"]},
                                                    {"name": "date_creation_enr", "type" : ["null","long"]},
                                                    {"name": "trt_maj_enr", "type" : ["null","string"]},
                                                    {"name": "date_maj_enr", "type" : ["null","long"]},
                                                    {"name": "flag_envoi_ca", "type" : ["null","string"]},
                                                    {"name": "date_envoi_ca", "type" : ["null","long"]},
                                                    {"name": "flag_transaction_recu", "type" : ["null","string"]},
                                                    {"name": "id_transaction", "type" : ["null","long"]},
                                                    {"name": "flag_histo_ca", "type" : ["null","string"]}
                                                ]
                                            }
                                        }
                                    },
                                    {"name": "histo_etat_colis",
                                        "doc": "on a 1 a N histo par colis",
                                        "type": {
                                            "type": "array",
                                            "items": {
                                                "type": "record",
                                                "name": "HistoEtatColis",
                                                "fields":[
                                                    {"name": "code_etat_colis", "type" : "string"},
                                                    {"name": "date_etat_colis", "type" : ["null","long"]},
                                                    {"name": "code_transport", "type" : "string"}
                                                ]
                                            }
                                        }
                                    }
                                ]
                            }
                        }
                    }
                ]
            }   
        }
    ]
}
``

[jamiefolson]

Hmm. Looks complicated but I don't see anything we had actively decided not to support for this version. Can you post a sample file that's failing?

[loicmathieu]

Hi,

Thanks to looking at this. I give a schema example in the issue. I'll attach an example AVRO file : commande2015-09-25 112458_9.avro.txt

We choose avro file format because it allow for complexe data structure (hierarchical one) so allow to have one record in one file for an order instead of 15 records split in 5 files ...

Regards,

Loïc