Create template for datacall

[cedricbriandgithub]

• [x] Annex 1
• [x] Annex 2
• [x] Annex 3
• [x] Annex 4
• [x] Annex 5
• [x] Annex 6
• [x] Annex 7
• [x] Annex 8
• [x] Annex 9 => new template for DCF, should refer to tables t_samplinginfo_sai t_qualitygroup_qug and t_biometrygroup_bug

related issue #104 #188

[cedricbriandgithub]

Annex 9 the tabs will be named sampling_info new_grouped_data updated_grouped_data deleted_grouped_data new_individual_data updated_individual_data deleted_individual_data

[cedricbriandgithub]

In series, explain distance to the sea (this is for Tea)

[cedricbriandgithub]

@elfunesto you add columns gr_id & grser_ser_id & gr_last_update & gr_dts_datasource in new_group_metrics ? They are not in the template, so this is inserted by your join am I right ? These should be left empty at this stage. I can adapt the load_function easily for this, just to check with you

Well in the following code the grser_ser_id are removed anyways, so it works fine, I'll correct my checks in load functions so as not to get a warning, so ignore my comments above

[elfunesto]

Normally, I  use the format specified in the root template, so if you prefer, we can just remove the column in the root template.  Currently, there is an instruction to leave it empty in the readme anyway.

[cedricbriandgithub]

Now this is the reason for my question, it should not be there.

As said above, you can ignore that comment, these columns will be overridden and ignored anyway.

[elfunesto]

ok, fixed: I was using the format of existing_metric (where gr_id is present) for both existing_metric and new_metric. I will upload and checking revised prefilled templates by the end of the morning

[Teica12]

In series, explain distance to the sea (this is for Tea)

It is already in the readme tab, but I have included it under series_info as well.

[cedricbriandgithub]

@elfunesto

das_dts_datasource is missing in updated_data and deleted data : what do we do ? Happy if we ignore it

[elfunesto]

it is not listed neither in existing_data. I don't think it is an issue in "updated_data" and "deleted_data" since it might be confusing for the data provider (does it refer to the datasource of the old data or of the current data call). I am not sure for existing data, however, having a similar structure between "existing_data" and the two other tabs facilitates copying/pasting. What's your opinion?

[cedricbriandgithub]

I think it is preferable to have it in existing data, since that information is present in the db, and copy it to updated and deleted data, currently we are generating the datasource in load series for new

      # for updated data and deleted data 
        if (grepl("data", sheet) & grepl("new", sheet)) {
            data_xls$das_dts_datasource <- datasource
        }       
        if (grepl("group", sheet)  & grepl("new", sheet)) {
            data_xls$gr_dts_datasource <- datasource
            data_xls$meg_dts_datasource <- datasource
        }
        if (grepl("individual", sheet)  & grepl("new", sheet)) {
            data_xls$gr_dts_datasource <- datasource
            data_xls$mei_dts_datasource <- datasource
        }

Later on we don't use this column in checking for modifications so it doesn't really matter what we have there, just that then we will be returning that info to the user.

    modified <- dplyr::anti_join(data_from_excel, data_from_base, 
            by = c("das_year", "das_value", "das_comment", "das_effort", "das_ser_id")
    )

So Im' changing my sheet to add it, (in the example FR you sent me).

[cedricbriandgithub]

@elfunesto Again another problem detected, please rename column gr_ser_id to grser_ser_id in updated and deleted data in series template

[cedricbriandgithub]

@elfunesto delete column gr_id in new group metrics, it is not created yet please also delete column grser_ser_id in new data it has no use, and risk to create errors. It is not present (no fiser_ser_id) in new_individual metrics