search

e-mission / op-admin-dashboard

An admin/deployer dashboard for the NREL OpenPATH platform
0 stars 8 forks source link

Script to automate appending tasks in JSON #75

Open niccolopaganini opened 10 months ago

niccolopaganini commented 10 months ago

Subset issue #70 created by @achasmita

Task that needs to be done from my end:

niccolopaganini commented 10 months ago

Sampling with this to see how it is being appended

data_to_append = {
    "key1": "val1",
    "key2": "val2"
}
niccolopaganini commented 10 months ago

update: I checked ca-ebike.nrel-op.json, stage-study.nrel-op.json, and denver-casr.nrel-op.json randomly to see if the append process works and they seem to append successfully.

niccolopaganini commented 10 months ago

let me try nested now:

"key_ob": {
        "key1": val1,
        "key2": val2
    }
niccolopaganini commented 10 months ago

Sampling with this to see how it is being appended

data_to_append = {
    "key1": "val1",
    "key2": "val2"
}

This is how the output looks like:

{
    "version": 1,
    "ts": 1655143472,
    "server": {
        "connectUrl": "https://ca-ebike-openpath.nrel.gov/api/",
        "aggregate_call_auth": "user_only"
    },
    "opcode": {
        "autogen": false,
        "subgroups": [
            "test",
            "default"
        ]
    },
    "intro": {
        "program_or_study": "program",
        "start_month": "05",
        "start_year": "2023",
        "mode_studied": "e-bike",
        "program_admin_contact": "dtfitch@ucdavis.edu, 530-601-7624",
        "deployment_partner_name": "University of California Davis",
        "translated_text": {
            "en": {
                "deployment_partner_name": "University of California, Davis",
                "deployment_name": "California E-bike Incentive Program",
                "summary_line_1": "The California Air Resources Board (CARB) aims to incentivize people to adopt energy-efficient, non-polluting modes of transportation and maximize air quality benefits.",
                "summary_line_2": "In this program, it is providing e-bikes to low-income individuals in communities around the state.",
                "summary_line_3": "If you are a program Participant, this app can help you log your trips and compete against others to benefit our environment.",
                "short_textual_description": " As e-bike incentive programs are spreading, NREL would like to experiment with an automated trip tracking solution for understanding travel behavior. This program evaluation can allow NREL to understand variations in travel across centers, days of the week and months of the year without complex, time-consuming surveys.",
                "why_we_collect": "The data collected during the program will be used to understand how e-bikes are used including when and how often as well as distance travelled. This in turn will aid in the estimation of GHG reductio from such incentive programs and new technologies.",
                "research_questions": [
                    "Travel behavior modeling to support cost-benefit analyses of various potential incentive programs.",
                    "The impact of e-bikes on transportation energy efficiency and GHG emissions."
                ]
            },
            "es": {
                "deployment_partner_name": "Universidad de California Davis",
                "deployment_name": "Programa de Incentivos para Bicicletas El\u00e9ctricas de California",
                "summary_line_1": "El Consejo de Recursos del Aire de California (CARB) apunta a incentivar a personas adoptar modas de transportacion energ\u00eda eficientes y no contaminantes y a maximizar los beneficios del calidad del aire.",
                "summary_line_2": "En este programa, proporciona bicicletas el\u00e9ctricas a personas de bajos ingresos en comunidades de todo el estado.",
                "summary_line_3": "Si es un participante del programa, esta aplicaci\u00f3n puede ayudarlo a registrar sus viajes y competir contra otros para beneficiar nuestro medio ambiente.",
                "short_textual_description": "Como programas de incentivos para bicicletas el\u00e9ctiracas est\u00e1n extendiendo, NREL quiere experimentar con una soluci\u00f3n automatizada de seguimiento de viajes para entender comportamiento de viaje. El evaluaci\u00f3n del programa se puede permitir a NREL entender variaciones de viaje tras centros, d\u00edas del semana y meses del a\u00f1o sin encuestas complejos que cuestan mucho tiempo.",
                "why_we_collect": "Los datos recopilados durante el programa se utilizar\u00e1n para comprender c\u00f3mo se utilizan las bicicletas el\u00e9ctricas, incluido cu\u00e1ndo y con qu\u00e9 frecuencia, as\u00ed como la distancia recorrida. Esto, a su vez, ayudar\u00e1 en la estimaci\u00f3n de la reducci\u00f3n de GEI de dichos programas de incentivos y nuevas tecnolog\u00edas. ",
                "research_questions": [
                    "Modelado del comportamiento de viaje para apoyar el an\u00e1lisis de costo-beneficio de varios programas de incentivos potenciales.",
                    " El impacto de las bicicletas el\u00e9ctricas en la eficiencia energ\u00e9tica del transporte y las emisiones de GEI."
                ]
            }
        }
    },
    "survey_info": {
        "surveys": {
            "UserProfileSurvey": {
                "formPath": "json/demo-survey-v2.json",
                "version": 1,
                "compatibleWith": 1,
                "dataKey": "manual/demographic_survey",
                "labelTemplate": {
                    "en": "Answered",
                    "es": "Contestada"
                }
            }
        },
        "trip-labels": "MULTILABEL"
    },
    "display_config": {
        "use_imperial": true
    },
    "metrics": {
        "include_test_users": true
    },
    "profile_controls": {
        "support_upload": false,
        "trip_end_notification": false
    },
    "admin_dashboard": {
        "overview_users": true,
        "overview_active_users": true,
        "overview_trips": true,
        "overview_signup_trends": true,
        "overview_trips_trend": true,
        "data_uuids": true,
        "data_trips": true,
        "data_trips_columns_exclude": [
            "data.start_loc.coordinates",
            "data.end_loc.coordinates"
        ],
        "additional_trip_columns": [],
        "data_uuids_columns_exclude": [],
        "token_generate": false,
        "token_prefix": "nrelop",
        "map_heatmap": true,
        "map_bubble": true,
        "map_trip_lines": false,
        "push_send": false,
        "options_uuids": true,
        "options_emails": true
    },
    "key1": "val1",
    "key2": "val2"
}
niccolopaganini commented 10 months ago

Seems to work fine. Output looks something like this (for ca-ebike.nrel-op.json):

{
    "version": 1,
    "ts": 1655143472,
    "server": {
        "connectUrl": "https://ca-ebike-openpath.nrel.gov/api/",
        "aggregate_call_auth": "user_only"
    },
    "opcode": {
        "autogen": false,
        "subgroups": [
            "test",
            "default"
        ]
    },
    "intro": {
        "program_or_study": "program",
        "start_month": "05",
        "start_year": "2023",
        "mode_studied": "e-bike",
        "program_admin_contact": "dtfitch@ucdavis.edu, 530-601-7624",
        "deployment_partner_name": "University of California Davis",
        "translated_text": {
            "en": {
                "deployment_partner_name": "University of California, Davis",
                "deployment_name": "California E-bike Incentive Program",
                "summary_line_1": "The California Air Resources Board (CARB) aims to incentivize people to adopt energy-efficient, non-polluting modes of transportation and maximize air quality benefits.",
                "summary_line_2": "In this program, it is providing e-bikes to low-income individuals in communities around the state.",
                "summary_line_3": "If you are a program Participant, this app can help you log your trips and compete against others to benefit our environment.",
                "short_textual_description": " As e-bike incentive programs are spreading, NREL would like to experiment with an automated trip tracking solution for understanding travel behavior. This program evaluation can allow NREL to understand variations in travel across centers, days of the week and months of the year without complex, time-consuming surveys.",
                "why_we_collect": "The data collected during the program will be used to understand how e-bikes are used including when and how often as well as distance travelled. This in turn will aid in the estimation of GHG reductio from such incentive programs and new technologies.",
                "research_questions": [
                    "Travel behavior modeling to support cost-benefit analyses of various potential incentive programs.",
                    "The impact of e-bikes on transportation energy efficiency and GHG emissions."
                ]
            },
            "es": {
                "deployment_partner_name": "Universidad de California Davis",
                "deployment_name": "Programa de Incentivos para Bicicletas El\u00e9ctricas de California",
                "summary_line_1": "El Consejo de Recursos del Aire de California (CARB) apunta a incentivar a personas adoptar modas de transportacion energ\u00eda eficientes y no contaminantes y a maximizar los beneficios del calidad del aire.",
                "summary_line_2": "En este programa, proporciona bicicletas el\u00e9ctricas a personas de bajos ingresos en comunidades de todo el estado.",
                "summary_line_3": "Si es un participante del programa, esta aplicaci\u00f3n puede ayudarlo a registrar sus viajes y competir contra otros para beneficiar nuestro medio ambiente.",
                "short_textual_description": "Como programas de incentivos para bicicletas el\u00e9ctiracas est\u00e1n extendiendo, NREL quiere experimentar con una soluci\u00f3n automatizada de seguimiento de viajes para entender comportamiento de viaje. El evaluaci\u00f3n del programa se puede permitir a NREL entender variaciones de viaje tras centros, d\u00edas del semana y meses del a\u00f1o sin encuestas complejos que cuestan mucho tiempo.",
                "why_we_collect": "Los datos recopilados durante el programa se utilizar\u00e1n para comprender c\u00f3mo se utilizan las bicicletas el\u00e9ctricas, incluido cu\u00e1ndo y con qu\u00e9 frecuencia, as\u00ed como la distancia recorrida. Esto, a su vez, ayudar\u00e1 en la estimaci\u00f3n de la reducci\u00f3n de GEI de dichos programas de incentivos y nuevas tecnolog\u00edas. ",
                "research_questions": [
                    "Modelado del comportamiento de viaje para apoyar el an\u00e1lisis de costo-beneficio de varios programas de incentivos potenciales.",
                    " El impacto de las bicicletas el\u00e9ctricas en la eficiencia energ\u00e9tica del transporte y las emisiones de GEI."
                ]
            }
        }
    },
    "survey_info": {
        "surveys": {
            "UserProfileSurvey": {
                "formPath": "json/demo-survey-v2.json",
                "version": 1,
                "compatibleWith": 1,
                "dataKey": "manual/demographic_survey",
                "labelTemplate": {
                    "en": "Answered",
                    "es": "Contestada"
                }
            }
        },
        "trip-labels": "MULTILABEL"
    },
    "display_config": {
        "use_imperial": true
    },
    "metrics": {
        "include_test_users": true
    },
    "profile_controls": {
        "support_upload": false,
        "trip_end_notification": false
    },
    "admin_dashboard": {
        "overview_users": true,
        "overview_active_users": true,
        "overview_trips": true,
        "overview_signup_trends": true,
        "overview_trips_trend": true,
        "data_uuids": true,
        "data_trips": true,
        "data_trips_columns_exclude": [
            "data.start_loc.coordinates",
            "data.end_loc.coordinates"
        ],
        "additional_trip_columns": [],
        "data_uuids_columns_exclude": [],
        "token_generate": false,
        "token_prefix": "nrelop",
        "map_heatmap": true,
        "map_bubble": true,
        "map_trip_lines": false,
        "push_send": false,
        "options_uuids": true,
        "options_emails": true
    },
    "key1": "val1",
    "key2": "val2",
    "key_ob": {
        "key1": "val1",
        "key2": "val2"
    }
}
niccolopaganini commented 10 months ago

I will submit an initial PR and then modify the "hard-coded" part to substitute the appropriate value

shankari commented 10 months ago

@niccolopaganini as I indicated in https://github.com/e-mission/op-admin-dashboard/issues/70#issuecomment-1744049354

the new key to add is data_trajectories_columns_exclude: ['user_id'], similar to the current data_trips_columns_exclude: key

Can you please update the testing done with it?

niccolopaganini commented 10 months ago

On it

niccolopaganini commented 10 months ago

Okay, my computer is lagging super bad and I am unable to work continuously. I already restarted it a couple of times but the issue is still persisting.

Working on the above issue

niccolopaganini commented 10 months ago

Addressing the top-level comment on PR (https://github.com/e-mission/nrel-openpath-deploy-configs/pull/40#discussion_r1347905144) -> Now, they're under admin_dashboard for all the JSON files present inside config

niccolopaganini commented 10 months ago

Pushing all the changes now...

niccolopaganini commented 10 months ago

Listing program names for convenience:

  1. ca-ebike.nrel-op.json
  2. stage-program.nrel-op.json
  3. stage-study.nrel-op.json
  4. uprm-civic.nrel-op.json
  5. uprm-nicr.nrel-op.json
  6. usaid-laos-ev.nrel-op.json
  7. uue.nrel-op.json
  8. washingtoncommons.nrel-op.json
niccolopaganini commented 10 months ago

Before I add what I tried, I also noted the weird indentation/ syntax issue occurring to the kv pair that we are trying to append. With this information, I noticed that this issue was only occurring in keys that have any object array pair. With that information, here's what I did:

  1. Augment the parent program to just include this: separators=(',', ':') ❌

  2. I thought appending might be the issue here; so I copied the whole chunk (chunk below for reference) and tried to replace the existing chunk (idk for some reason/ 4:30 AM me thought this might work) ❌ 

    "admin_dashboard": {
    "overview_users": true,
    "overview_active_users": true,
    "overview_trips": true,
    "overview_signup_trends": true,
    "overview_trips_trend": true,
    "data_uuids": true,
    "data_trips": true,
    "data_trips_columns_exclude": [
        "data.start_loc.coordinates",
        "data.end_loc.coordinates"
    ],
    "additional_trip_columns": [],
    "data_uuids_columns_exclude": [],
    "token_generate": false,
    "token_prefix": "nrelop",
    "map_heatmap": true,
    "map_bubble": true,
    "map_trip_lines": false,
    "push_send": false,
    "options_uuids": true,
    "options_emails": true,
    "data_trajectories_columns_exclude": ["user_id"]

  3. I thought of using the information I found above and incorporate it in a script addressing the weird spacing issue ❌

  4. With the latest push, I was only able to fix the last part because I was able to isolate it (because that's the part that is being appended to the list) 🤕

niccolopaganini commented 10 months ago

I found this: https://github.com/microsoft/TypeScript/issues/15034#issuecomment-337465163 @shankari do you think I might be facing a similar issue for square brackets?

shankari commented 10 months ago

@niccolopaganini

I found this: https://github.com/microsoft/TypeScript/issues/15034#issuecomment-337465163 @shankari do you think I might be facing a similar issue for square brackets?

No, because that is typescript and this is JSON.

With this information, I noticed that this issue was only occurring in keys that have any object array pair. With that information, here's what I did:

That is a good point. As I said in the PR review:

If you can't figure out the JSON dump arguments to enable this, feel free to use jq to edit the files directly instead. If there is a reason why this doesn't work, please document it

So, I would like to see:

If it is not possible to fix this, the original JSON format may not be compatible with JSON dump in the first place - much of that JSON was hand-created, so that would not be super suprising.

If you can make the argument for that, you can submit two PRs: