Indonesia Rollout Update

[tm-jc-nacpil]

This notebook updates the Indonesia rollout output and comparison

Changes

• Updated the model to use MinMax Scaler
• Fix bug wherein the nightttime lights are zero after feature processing
• Fix bug wherein the model predictions are attached to the input grids out of order

Results

• Compared to the SUSENAS wealth index data, we achieved a rank correlaion of 0.72 when ranking the average wealth per adm2 level

[review-notebook-app[bot]]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:30Z ----------------------------------------------------------------

Line #1.    aoi_data = gpd.read_file(f'{rollout_date}-{country_code}-rollout-features.geojson')

Would recommend to revert the notebook to be completely runnable from top to bottom without relying on loading data from intermediate files.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:31Z ----------------------------------------------------------------

Line #3.    q_decimal = q / 100

Since we decided not to do this anymore, would recommend just deleting these quantile configs to reeduce complexity.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:32Z ----------------------------------------------------------------

Line #4.        max_val = features[[col]].quantile(q_decimal)

Same comment re: quantile. Let's just remove it.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:33Z ----------------------------------------------------------------

Line #1.    # Uncomment this cell and run to save a local copy of the scaled features

Since you said uncomment, might want to make the code commented by default haha

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:34Z ----------------------------------------------------------------

Please remove this whole part; our asset index does not directly relate to income and socio-economic class.

tm-jc-nacpil commented on 2023-03-24T07:25:04Z ----------------------------------------------------------------

Hi alron, do you mean removing even the binning step entirely or just the markdown explanation?

alronlam commented on 2023-03-24T08:35:59Z ----------------------------------------------------------------

I meant just the markdown explanation (and the split-quintile parts). But retain the quintiles.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:35Z ----------------------------------------------------------------

Delete this whole split-quintile category section too.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:36Z ----------------------------------------------------------------

Line #2.    rollout_aoi.to_file(f'{rollout_date}-{country_code}-rollout-output-minmax-q{q}.geojson', driver='GeoJSON', index=False)

1. What is q?

2. Double checking, this file also contain population counts right?

3. Noting for code clean-up sprint, we might want to refactor so that we save a copy of:

1. the rollout aoi + scaled features + predicted wealth (raw and bins)?
2. the rollout aoi + raw features + predicted wealth (raw and bins)?

This will make any further EDAs much faster, and allows anybody to do so without having to re-run Indonesia.

Illustrative example based on the single country notebooks:

# Join back the features
rollout_output_with_unscaled_features = rollout_aoi.join(features)
<save this file>

<do the same for scaled>

tm-jc-nacpil commented on 2023-03-24T08:32:17Z ----------------------------------------------------------------

Done!

tm-jc-nacpil commented 1 year ago

Hi alron, do you mean removing even the binning step entirely or just the markdown explanation?

---

View entire conversation on ReviewNB

tm-jc-nacpil commented 1 year ago

Done!

---

View entire conversation on ReviewNB

alronlam commented 1 year ago

I meant just the markdown explanation (and the split-quintile parts). But retain the quintiles.

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:31Z ----------------------------------------------------------------

Line #3.    q_decimal = q / 100

Since we decided not to do this anymore, would recommend just deleting these quantile configs to reeduce complexity.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:32Z ----------------------------------------------------------------

Line #4.        max_val = features[[col]].quantile(q_decimal)

Same comment re: quantile. Let's just remove it.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:33Z ----------------------------------------------------------------

Line #1.    # Uncomment this cell and run to save a local copy of the scaled features

Since you said uncomment, might want to make the code commented by default haha

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:34Z ----------------------------------------------------------------

Please remove this whole part; our asset index does not directly relate to income and socio-economic class.

tm-jc-nacpil commented on 2023-03-24T07:25:04Z ----------------------------------------------------------------

Hi alron, do you mean removing even the binning step entirely or just the markdown explanation?

alronlam commented on 2023-03-24T08:35:59Z ----------------------------------------------------------------

I meant just the markdown explanation (and the split-quintile parts). But retain the quintiles.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:35Z ----------------------------------------------------------------

Delete this whole split-quintile category section too.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:36Z ----------------------------------------------------------------

Line #2.    rollout_aoi.to_file(f'{rollout_date}-{country_code}-rollout-output-minmax-q{q}.geojson', driver='GeoJSON', index=False)

1. What is q?

2. Double checking, this file also contain population counts right?

3. Noting for code clean-up sprint, we might want to refactor so that we save a copy of:

1. the rollout aoi + scaled features + predicted wealth (raw and bins)?
2. the rollout aoi + raw features + predicted wealth (raw and bins)?

This will make any further EDAs much faster, and allows anybody to do so without having to re-run Indonesia.

Illustrative example based on the single country notebooks:

# Join back the features
rollout_output_with_unscaled_features = rollout_aoi.join(features)
<save this file>

<do the same for scaled>

tm-jc-nacpil commented on 2023-03-24T08:32:17Z ----------------------------------------------------------------

Done!

tm-jc-nacpil commented 1 year ago

Hi alron, do you mean removing even the binning step entirely or just the markdown explanation?

---

View entire conversation on ReviewNB

tm-jc-nacpil commented 1 year ago

Done!

---

View entire conversation on ReviewNB

alronlam commented 1 year ago

I meant just the markdown explanation (and the split-quintile parts). But retain the quintiles.

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:32Z ----------------------------------------------------------------

Line #4.        max_val = features[[col]].quantile(q_decimal)

Same comment re: quantile. Let's just remove it.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:33Z ----------------------------------------------------------------

Line #1.    # Uncomment this cell and run to save a local copy of the scaled features

Since you said uncomment, might want to make the code commented by default haha

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:34Z ----------------------------------------------------------------

Please remove this whole part; our asset index does not directly relate to income and socio-economic class.

tm-jc-nacpil commented on 2023-03-24T07:25:04Z ----------------------------------------------------------------

Hi alron, do you mean removing even the binning step entirely or just the markdown explanation?

alronlam commented on 2023-03-24T08:35:59Z ----------------------------------------------------------------

I meant just the markdown explanation (and the split-quintile parts). But retain the quintiles.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:35Z ----------------------------------------------------------------

Delete this whole split-quintile category section too.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:36Z ----------------------------------------------------------------

Line #2.    rollout_aoi.to_file(f'{rollout_date}-{country_code}-rollout-output-minmax-q{q}.geojson', driver='GeoJSON', index=False)

1. What is q?

2. Double checking, this file also contain population counts right?

3. Noting for code clean-up sprint, we might want to refactor so that we save a copy of:

1. the rollout aoi + scaled features + predicted wealth (raw and bins)?
2. the rollout aoi + raw features + predicted wealth (raw and bins)?

This will make any further EDAs much faster, and allows anybody to do so without having to re-run Indonesia.

Illustrative example based on the single country notebooks:

# Join back the features
rollout_output_with_unscaled_features = rollout_aoi.join(features)
<save this file>

<do the same for scaled>

tm-jc-nacpil commented on 2023-03-24T08:32:17Z ----------------------------------------------------------------

Done!

tm-jc-nacpil commented 1 year ago

Hi alron, do you mean removing even the binning step entirely or just the markdown explanation?

---

View entire conversation on ReviewNB

tm-jc-nacpil commented 1 year ago

Done!

---

View entire conversation on ReviewNB

alronlam commented 1 year ago

I meant just the markdown explanation (and the split-quintile parts). But retain the quintiles.

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:33Z ----------------------------------------------------------------

Line #1.    # Uncomment this cell and run to save a local copy of the scaled features

Since you said uncomment, might want to make the code commented by default haha

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:34Z ----------------------------------------------------------------

Please remove this whole part; our asset index does not directly relate to income and socio-economic class.

tm-jc-nacpil commented on 2023-03-24T07:25:04Z ----------------------------------------------------------------

Hi alron, do you mean removing even the binning step entirely or just the markdown explanation?

alronlam commented on 2023-03-24T08:35:59Z ----------------------------------------------------------------

I meant just the markdown explanation (and the split-quintile parts). But retain the quintiles.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:35Z ----------------------------------------------------------------

Delete this whole split-quintile category section too.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:36Z ----------------------------------------------------------------

Line #2.    rollout_aoi.to_file(f'{rollout_date}-{country_code}-rollout-output-minmax-q{q}.geojson', driver='GeoJSON', index=False)

1. What is q?

2. Double checking, this file also contain population counts right?

3. Noting for code clean-up sprint, we might want to refactor so that we save a copy of:

1. the rollout aoi + scaled features + predicted wealth (raw and bins)?
2. the rollout aoi + raw features + predicted wealth (raw and bins)?

This will make any further EDAs much faster, and allows anybody to do so without having to re-run Indonesia.

Illustrative example based on the single country notebooks:

# Join back the features
rollout_output_with_unscaled_features = rollout_aoi.join(features)
<save this file>

<do the same for scaled>

tm-jc-nacpil commented on 2023-03-24T08:32:17Z ----------------------------------------------------------------

Done!

tm-jc-nacpil commented 1 year ago

Hi alron, do you mean removing even the binning step entirely or just the markdown explanation?

---

View entire conversation on ReviewNB

tm-jc-nacpil commented 1 year ago

Done!

---

View entire conversation on ReviewNB

alronlam commented 1 year ago

I meant just the markdown explanation (and the split-quintile parts). But retain the quintiles.

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:34Z ----------------------------------------------------------------

Please remove this whole part; our asset index does not directly relate to income and socio-economic class.

tm-jc-nacpil commented on 2023-03-24T07:25:04Z ----------------------------------------------------------------

Hi alron, do you mean removing even the binning step entirely or just the markdown explanation?

alronlam commented on 2023-03-24T08:35:59Z ----------------------------------------------------------------

I meant just the markdown explanation (and the split-quintile parts). But retain the quintiles.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:35Z ----------------------------------------------------------------

Delete this whole split-quintile category section too.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:36Z ----------------------------------------------------------------

Line #2.    rollout_aoi.to_file(f'{rollout_date}-{country_code}-rollout-output-minmax-q{q}.geojson', driver='GeoJSON', index=False)

1. What is q?

2. Double checking, this file also contain population counts right?

3. Noting for code clean-up sprint, we might want to refactor so that we save a copy of:

1. the rollout aoi + scaled features + predicted wealth (raw and bins)?
2. the rollout aoi + raw features + predicted wealth (raw and bins)?

This will make any further EDAs much faster, and allows anybody to do so without having to re-run Indonesia.

Illustrative example based on the single country notebooks:

# Join back the features
rollout_output_with_unscaled_features = rollout_aoi.join(features)
<save this file>

<do the same for scaled>

tm-jc-nacpil commented on 2023-03-24T08:32:17Z ----------------------------------------------------------------

Done!

tm-jc-nacpil commented 1 year ago

Hi alron, do you mean removing even the binning step entirely or just the markdown explanation?

---

View entire conversation on ReviewNB

tm-jc-nacpil commented 1 year ago

Done!

---

View entire conversation on ReviewNB

alronlam commented 1 year ago

I meant just the markdown explanation (and the split-quintile parts). But retain the quintiles.

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:35Z ----------------------------------------------------------------

Delete this whole split-quintile category section too.

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:36Z ----------------------------------------------------------------

Line #2.    rollout_aoi.to_file(f'{rollout_date}-{country_code}-rollout-output-minmax-q{q}.geojson', driver='GeoJSON', index=False)

1. What is q?

2. Double checking, this file also contain population counts right?

3. Noting for code clean-up sprint, we might want to refactor so that we save a copy of:

1. the rollout aoi + scaled features + predicted wealth (raw and bins)?
2. the rollout aoi + raw features + predicted wealth (raw and bins)?

This will make any further EDAs much faster, and allows anybody to do so without having to re-run Indonesia.

Illustrative example based on the single country notebooks:

# Join back the features
rollout_output_with_unscaled_features = rollout_aoi.join(features)
<save this file>

<do the same for scaled>

tm-jc-nacpil commented on 2023-03-24T08:32:17Z ----------------------------------------------------------------

Done!

tm-jc-nacpil commented 1 year ago

Hi alron, do you mean removing even the binning step entirely or just the markdown explanation?

---

View entire conversation on ReviewNB

tm-jc-nacpil commented 1 year ago

Done!

---

View entire conversation on ReviewNB

alronlam commented 1 year ago

I meant just the markdown explanation (and the split-quintile parts). But retain the quintiles.

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

alronlam commented on 2023-03-24T07:03:36Z ----------------------------------------------------------------

Line #2.    rollout_aoi.to_file(f'{rollout_date}-{country_code}-rollout-output-minmax-q{q}.geojson', driver='GeoJSON', index=False)

1. What is q?

2. Double checking, this file also contain population counts right?

3. Noting for code clean-up sprint, we might want to refactor so that we save a copy of:

1. the rollout aoi + scaled features + predicted wealth (raw and bins)?
2. the rollout aoi + raw features + predicted wealth (raw and bins)?

This will make any further EDAs much faster, and allows anybody to do so without having to re-run Indonesia.

Illustrative example based on the single country notebooks:

# Join back the features
rollout_output_with_unscaled_features = rollout_aoi.join(features)
<save this file>

<do the same for scaled>

tm-jc-nacpil commented on 2023-03-24T08:32:17Z ----------------------------------------------------------------

Done!

tm-jc-nacpil commented 1 year ago

Hi alron, do you mean removing even the binning step entirely or just the markdown explanation?

---

View entire conversation on ReviewNB

tm-jc-nacpil commented 1 year ago

Done!

---

View entire conversation on ReviewNB

alronlam commented 1 year ago

I meant just the markdown explanation (and the split-quintile parts). But retain the quintiles.

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[tm-jc-nacpil]

Hi alron, do you mean removing even the binning step entirely or just the markdown explanation?

---

View entire conversation on ReviewNB

[tm-jc-nacpil]

Done!

---

View entire conversation on ReviewNB

[alronlam]

I meant just the markdown explanation (and the split-quintile parts). But retain the quintiles.

---

View entire conversation on ReviewNB