Does not restore app_settings correctly, plus gets ridiculously bloated

[abbyad]

I figure that these are related issues so posting them together, but feel free to separate.

• On a fresh instance I restored a new test version of app_settings and the resulting app_settings maintains the old rules, not the new ones.
• Also, got hundreds of duplicates of the old properties, which bogs down app_settings and the whole app.

Here is the app_settings I was trying to restore. It is valid json but may have app_settings structure errors. This may be related to #521

{
    "locale": "en",
    "muvuku_webapp_url": "/muvuku-webapp/_design/muvuku-webapp/_rewrite/?_embed_mode=1&_locale=en#/generic-anc.json",
    "reported_date_format": "MMM hh:mm",
    "forms_only_mode": true,
    "gateway_number": "",
    "outgoing_phone_filters": [{
        "match": "",
        "replace": ""
    }],
    "schedule_morning_hours": 6,
    "schedule_evening_hours": 13,
    "synthetic_date": "",
    "id_format": "1111",
    "contact_display_short": "clinic.name",
    "kujua-reporting": [{
        "code": "",
        "reporting_freq": "weekly"
    }],
    "alerts": [{
        "form": "",
        "condition": "",
        "message": "",
        "recipient": ""
    }],
    "reminders": [{
        "form": "",
        "text_expression": "",
        "cron": "",
        "mute_after_form_for": "",
        "message": ""
    }],
    "registrations": [{
        "form": "R",
        "events": [{
            "name": "on_create",
            "trigger": "add_patient_id",
            "params": "",
            "bool_expr": ""
        },
        {
            "name": "on_create",
            "trigger": "add_expected_date",
            "params": "lmp_date",
            "bool_expr": "doc.last_menstrual_period"
        },
        {
            "name": "on_create",
            "trigger": "assign_schedule",
            "params": "ANC Reminders LMP",
            "bool_expr": "doc.last_menstrual_period"
        },
        {
            "name": "on_create",
            "trigger": "assign_schedule",
            "params": "ANC Reminders",
            "bool_expr": "doc.last_menstrual_period === ''"
        }],
        "validations": {
            "join_responses": false,
            "list": [{
                "property": "patient_name",
                "rule": "lenMin(1) && lenMax(30)",
                "message": "{{#patient_name}}The registration format is incorrect, ensure the message starts with R followed by space and the mother's name (maximum of 30 characters).{{/patient_name}}{{^patient_name}}The registration format is incorrect. ensure the message starts with R followed by space and the mother's name.{{/patient_name}}."
            },
            {
                "property": "last_menstrual_period",
                "rule": "min(0) ? max(42) : otherMin('age_in_years', 0) && otherMax('age_in_years', 60)",
                "message": "The registration format for '{{patient_name}}' is incorrect, please ensure that it is a number between 0 and 42."
            }]
        },
        "messages": [{
            "message": "Thank you {{contact.name}} for registering {{patient_name}}. Pregnancy ID is {{patient_id}}.",
            "recipient": "reporting_unit",
            "locale": "en"
        }]
    }],
    "patient_reports": [{
        "form": "V",
        "silence_type": "ANC Reminders, ANC Reminders LMP",
        "silence_for": "25 days",
        "validations": {
            "join_responses": false,
            "list": [{
                "property": "patient_id",
                "rule": "regex('^[0-9]{5}$')",
                "message": "{{#patient_id}}The ID number submitted is incorrect, it should be 5 numbers, please submit a valid ID.{{/patient_id}}{{^patient_id}}The message format is incorrect. Please ensure send a message start with V followed by space then the woman's ID.{{/patient_id}}"
            }]
        },
        "messages": [{
            "message": "Thank you {{contact.name}}, visit for {{patient_name}} ({{patient_id}}) has been recorded.",
            "event_type": "report_accepted",
            "recipient": "reporting_unit"
        },
        {
            "message": "No woman with ID number '{{patient_id}}' found. Verify the ID and resend the message.",
            "event_type": "registration_not_found",
            "recipient": "reporting_unit"
        }]
    },
    {
        "form": "D",
        "silence_type": "ANC Reminders",
        "silence_for": "",
        "validations": {
            "join_responses": false,
            "list": [{
                "property": "patient_id",
                "rule": "regex('^[0-9]{5}$')",
                "message": "{{#patient_id}}The ID number submitted is incorrect, it should be 5 numbers, please submit a valid ID.{{/patient_id}}{{^patient_id}}The message format is incorrect. Please ensure send a message start with V followed by space then the woman's ID.{{/patient_id}}"
            },
            {
                "property": "delivery_code",
                "rule": "regex('^(F|f|S|s|Ns|NS|ns|nS)$')",
                "message": "{{#delivery_code}}The delivery code was not understood. Please use code 'F' for facility, 'S' for homebirth with Skilled Birth Attendant, 'NS' for birth without SBA.{{/delivery_code}}{{^delivery_code}}The delivery code was missing. Please use code 'F' for facility, 'S' for homebirth with Skilled Birth Attendant, 'NS' for birth without SBA.{{/delivery_code}}"
            }]
        },
        "messages": [{
            "message": "Thank you {{contact.name}} for reporting on {{patient_name}} ({{patient_id}}). No further notifications will be sent for this patient.",
            "event_type": "report_accepted",
            "recipient": "reporting_unit"
        },
        {
            "message": "No woman with ID number '{{patient_id}}' found. Verify the ID and resend the message.",
            "event_type": "registration_not_found",
            "recipient": "reporting_unit"
        }]
    },
    {
        "form": "F",
        "silence_type": "",
        "silence_for": "",
        "validations": {
            "join_responses": false,
            "list": [{
                "property": "patient_id",
                "rule": "regex('^[0-9]{5}$')",
                "message": "{{#patient_id}}The ID number submitted is incorrect, it should be 5 numbers, please submit a valid ID.{{/patient_id}}{{^patient_id}}The message format is incorrect. Please ensure send a message start with V followed by space then the woman's ID.{{/patient_id}}"
            }]
        },
        "messages": [{
            "message": "We have added {{patient_name}} ({{patient_id}}) to the flagged pregnancy list. If this is an emergency, please refer her to the clinic immediately.",
            "event_type": "report_accepted",
            "recipient": "reporting_unit"
        },
        {
            "message": "No woman with ID number '{{patient_id}}' found. Verify the ID and resend the message.",
            "event_type": "registration_not_found",
            "recipient": "reporting_unit"
        }]
    }],
    "schedules": [{
        "name": "ANC Reminders LMP",
        "start_from": "lmp_date",
        "messages": [{
            "message": "Please remind {{patient_name}} ({{patient_id}}) to visit health facility for ANC visit this week. When she does let us know with 'V {{patient_id}}'. Thanks!",
            "group": 1,
            "offset": "12 weeks",
            "send_day": "monday",
            "send_time": "09:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Did {{patient_name}} attend her ANC visit? When she does, respond with 'V {{patient_id}}'. Thank you!",
            "group": 1,
            "offset": "12 weeks",
            "send_day": "monday",
            "send_time": "10:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Please remind {{patient_name}} ({{patient_id}}) to visit health facility for ANC visit this week. When she does let us know with 'V {{patient_id}}'. Thanks!",
            "group": 2,
            "offset": "24 weeks",
            "send_day": "monday",
            "send_time": "09:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Did {{patient_name}} attend her ANC visit? When she does, respond with 'V {{patient_id}}'. Thank you!",
            "group": 2,
            "offset": "24 weeks",
            "send_day": "monday",
            "send_time": "10:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Please remind {{patient_name}} ({{patient_id}}) to visit health facility for ANC visit this week. When she does let us know with 'V {{patient_id}}'. Thanks!",
            "group": 3,
            "offset": "32 weeks",
            "send_day": "monday",
            "send_time": "09:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Did {{patient_name}} attend her ANC visit? When she does, respond with 'V {{patient_id}}'. Thank you!",
            "group": 3,
            "offset": "32 weeks",
            "send_day": "monday",
            "send_time": "10:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Please remind {{patient_name}} ({{patient_id}}) to visit health facility for ANC visit this week. When she does let us know with 'V {{patient_id}}'. Thanks!",
            "group": 4,
            "offset": "36 weeks",
            "send_day": "monday",
            "send_time": "09:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Did {{patient_name}} attend her ANC visit? When she does, respond with 'V {{patient_id}}'. Thank you!",
            "group": 4,
            "offset": "36 weeks",
            "send_day": "monday",
            "send_time": "10:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Please remind {{patient_name}} ({{patient_id}}) to visit health facility for ANC visit this week. When she does let us know with 'V {{patient_id}}'. Thanks!",
            "group": 5,
            "offset": "40 weeks",
            "send_day": "monday",
            "send_time": "09:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Did {{patient_name}} attend her ANC visit? When she does, respond with 'V {{patient_id}}'. Thank you!",
            "group": 5,
            "offset": "40 weeks",
            "send_day": "monday",
            "send_time": "10:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Where did {{patient_name}} deliver? Respond with 'D F {{patient_id}}' for facility birth, 'D S {{patient_id}}' for home birth with SBA, 'D NS {{patient_id}}' for home birth.",
            "group": 6,
            "offset": "45 weeks",
            "send_day": "monday",
            "send_time": "11:00",
            "recipient": "reporting_unit",
            "locale": "en"
        }]
    },
    {
        "name": "ANC Reminders",
        "start_from": "reported_date",
        "messages": [{
            "message": "Please remind {{patient_name}} ({{patient_id}}) to visit health facility for ANC visit this week. When she does let us know with 'V {{patient_id}}'. Thanks!",
            "group": 1,
            "offset": "0 weeks",
            "send_day": "monday",
            "send_time": "09:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Did {{patient_name}} attend her ANC visit? When she does, respond with 'V {{patient_id}}'. Thank you!",
            "group": 1,
            "offset": "0 weeks",
            "send_day": "monday",
            "send_time": "10:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Please remind {{patient_name}} ({{patient_id}}) to visit health facility for ANC visit this week. When she does let us know with 'V {{patient_id}}'. Thanks!",
            "group": 2,
            "offset": "4 weeks",
            "send_day": "monday",
            "send_time": "09:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Did {{patient_name}} attend her ANC visit? When she does, respond with 'V {{patient_id}}'. Thank you!",
            "group": 2,
            "offset": "4 weeks",
            "send_day": "monday",
            "send_time": "10:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Please remind {{patient_name}} ({{patient_id}}) to visit health facility for ANC visit this week. When she does let us know with 'V {{patient_id}}'. Thanks!",
            "group": 3,
            "offset": "8 weeks",
            "send_day": "monday",
            "send_time": "09:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Did {{patient_name}} attend her ANC visit? When she does, respond with 'V {{patient_id}}'. Thank you!",
            "group": 3,
            "offset": "8 weeks",
            "send_day": "monday",
            "send_time": "10:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Please remind {{patient_name}} ({{patient_id}}) to visit health facility for ANC visit this week. When she does let us know with 'V {{patient_id}}'. Thanks!",
            "group": 4,
            "offset": "12 weeks",
            "send_day": "monday",
            "send_time": "09:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Did {{patient_name}} attend her ANC visit? When she does, respond with 'V {{patient_id}}'. Thank you!",
            "group": 4,
            "offset": "12 weeks",
            "send_day": "monday",
            "send_time": "10:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Please remind {{patient_name}} ({{patient_id}}) to visit health facility for ANC visit this week. When she does let us know with 'V {{patient_id}}'. Thanks!",
            "group": 5,
            "offset": "16 weeks",
            "send_day": "monday",
            "send_time": "09:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Did {{patient_name}} attend her ANC visit? When she does, respond with 'V {{patient_id}}'. Thank you!",
            "group": 5,
            "offset": "16 weeks",
            "send_day": "monday",
            "send_time": "10:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Please remind {{patient_name}} ({{patient_id}}) to visit health facility for ANC visit this week. When she does let us know with 'V {{patient_id}}'. Thanks!",
            "group": 6,
            "offset": "20 weeks",
            "send_day": "monday",
            "send_time": "09:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Did {{patient_name}} attend her ANC visit? When she does, respond with 'V {{patient_id}}'. Thank you!",
            "group": 6,
            "offset": "20 weeks",
            "send_day": "monday",
            "send_time": "10:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Please remind {{patient_name}} ({{patient_id}}) to visit health facility for ANC visit this week. When she does let us know with 'V {{patient_id}}'. Thanks!",
            "group": 7,
            "offset": "24 weeks",
            "send_day": "monday",
            "send_time": "09:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Did {{patient_name}} attend her ANC visit? When she does, respond with 'V {{patient_id}}'. Thank you!",
            "group": 7,
            "offset": "24 weeks",
            "send_day": "monday",
            "send_time": "10:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Please remind {{patient_name}} ({{patient_id}}) to visit health facility for ANC visit this week. When she does let us know with 'V {{patient_id}}'. Thanks!",
            "group": 8,
            "offset": "28 weeks",
            "send_day": "monday",
            "send_time": "09:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Did {{patient_name}} attend her ANC visit? When she does, respond with 'V {{patient_id}}'. Thank you!",
            "group": 8,
            "offset": "28 weeks",
            "send_day": "monday",
            "send_time": "10:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Please remind {{patient_name}} ({{patient_id}}) to visit health facility for ANC visit this week. When she does let us know with 'V {{patient_id}}'. Thanks!",
            "group": 9,
            "offset": "32 weeks",
            "send_day": "monday",
            "send_time": "09:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Did {{patient_name}} attend her ANC visit? When she does, respond with 'V {{patient_id}}'. Thank you!",
            "group": 9,
            "offset": "32 weeks",
            "send_day": "monday",
            "send_time": "10:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Please remind {{patient_name}} ({{patient_id}}) to visit health facility for ANC visit this week. When she does let us know with 'V {{patient_id}}'. Thanks!",
            "group": 10,
            "offset": "36 weeks",
            "send_day": "monday",
            "send_time": "09:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Did {{patient_name}} attend her ANC visit? When she does, respond with 'V {{patient_id}}'. Thank you!",
            "group": 10,
            "offset": "36 weeks",
            "send_day": "monday",
            "send_time": "10:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Please remind {{patient_name}} ({{patient_id}}) to visit health facility for ANC visit this week. When she does let us know with 'V {{patient_id}}'. Thanks!",
            "group": 11,
            "offset": "40 weeks",
            "send_day": "monday",
            "send_time": "09:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Did {{patient_name}} attend her ANC visit? When she does, respond with 'V {{patient_id}}'. Thank you!",
            "group": 11,
            "offset": "40 weeks",
            "send_day": "monday",
            "send_time": "10:00",
            "recipient": "reporting_unit",
            "locale": "en"
        },
        {
            "message": "Where did {{patient_name}} deliver? Respond with 'D F {{patient_id}}' for facility birth, 'D S {{patient_id}}' for home birth with SBA, 'D NS {{patient_id}}' for home birth.",
            "group": 12,
            "offset": "42 weeks",
            "send_day": "monday",
            "send_time": "11:00",
            "recipient": "reporting_unit",
            "locale": "en"
        }]
    }],
    "notifications": {
        "off_form": "STOP",
        "on_form": "START",
        "confirm_deactivation": false,
        "validations": [{
            "property": "patient_id",
            "rule": "regex('^[0-9]{5}$')",
            "message": "{{#patient_id}}The ID number submitted is incorrect, it should be 5 numbers, please submit a valid ID.{{/patient_id}}{{^patient_id}}The message format is incorrect. Please ensure send a message start with V followed by space then the woman's ID.{{/patient_id}}"
        }],
        "messages": [{
            "message": "Thank you {{contact.name}}, no further notifications regarding {{patient_name}} will be sent until you submit 'START {{patient_id}}'.",
            "event_type": "on_mute",
            "recipient": "reporting_unit"
        },
        {
            "message": "Thank you {{contact.name}}, record for {{patient_name}} ({{patient_id}}) has been reactivated. Notifications regarding this patient will resume.",
            "event_type": "on_unmute",
            "recipient": "reporting_unit"
        },
        {
            "message": "No patient with ID number '{{patient_number}}' found.",
            "event_type": "patient_not_found",
            "recipient": "reporting_unit"
        }]
    },
    "_version": "0.4.0-alpha.580"
}

[abbyad]

I forgot to add this screenshot. Notice the search found 430 results of the same validation text. If it is helpful I can also post a link to the resulting app_settings (1.2MB)

[garethbowen]

The _version parameter indicates this file was exported from version 0.4.0-alpha.580 but it's not valid in that version (the message components should be arrays of translated messages). Was this import file manually edited at all?

If I remove the version parameter then it appears to work as expected.

[abbyad]

Good find... I had manually added the version because it was not being restored otherwise. Does 'restore' migrate the app_settings yet? If not we should create an issue for that

[garethbowen]

Yes. Migration on restore was added at the same time as migration on update. So if you don't include the version then the migration runs.

[abbyad]

That migration was failing for me last week but will test again and report back.

[garethbowen]

We've fixed a couple of bugs in the migration last week, so you should be good to go

[abbyad]

Seen to work in 0.4.0-alpha.599