Atharva_Goals_and_Questions

[AtharvaBeesen]

Weekly Notebook Entry — Week 1

Quick Overview

List your top 3 tasks or objectives for this week:

Task 1: Reading up on GeoPandas to help with writing code. Task 2: Working with Albert to identify potential new parameters and subindices. Task 3: Writing up a first draft of code to incorporate these parameters.

Weekly Accomplishments

What tasks did you complete this week?

Looked over the codebase and familiarized myself with all the libraries and techniques used in developing the output from last semester. Also compiled a word document with detailed notes on the codebase if I need to understand something in the future.

Tasks are still ongoing? Current status?

Need to work with Albert to identify some more parameters and subindices in order to further develop the code.

Challenges and Learning

What was the biggest challenge you faced this week? How did you address it?

Attempting to understand new Python libraries and analysis techniques that I had not seen before (particularly working with geographic data). I addressed this challenge through watching YouTube tutorials and reading documentation on the libraries' official websites.

What's one new thing you learned or skill you improved?

Working with GeoJSON files!

Did you attend any team meetings? Key takeaways?

Attended a virtual Teams meeting with my team where we laid out a rough plan and assigned tasks for each of the team members - my goal was to familiarize myself with the codebase.

Reflection and Planning

Your progress this week?

(from Weekly Accomplishments) Looked over the codebase and familiarized myself with all the libraries and techniques used in developing the output from last semester. Also compiled a word document with detailed notes on the codebase if I need to understand something in the future. Will also be sending my Word document and a PowerPoint with more details about my work this week to our Teams groupchat.

Main focus for next week?

Making a plan regarding parameters and subindices to look back at when it's time to write some code! Also to watch a few more tutorials on GeoPandas.

Any resources you're looking for?

N/A

[AtharvaBeesen]

Weekly Notebook Entry — Week 2

Quick Overview

List your top 3 tasks or objectives for this week:

• Task 1: Think of new ways to update the statistic to make it easier to understand/more valid.
• Task 2: Plan out a method for transforming new/different data sources into GeoJson so we can update the statistic with new parameters.
• Task 3: Work with the rest of the team to identify these new parameters.
• (Bonus) Task 4: Figure out a way to display our data and statistics — our idea is a web app.

Weekly Accomplishments

What tasks did you complete this week?

Looked for ways to update our PEI statistic and tested a few different statistical approaches to improving the output values to be easier to comprehend (e.g. rank-based transformation and sigmoid transformation). Also created notes for ways to make the statistic more useful/valid/realistic (i.e., so that the statistic is not just relative to the city it is in, but to the whole country).

Tasks still ongoing? Current status?

Need to work with Albert to identify more parameters and subindices in order to further develop the code — idea generation is going well; we just need to finalize the exact subindices and APIs to use before integrating into the code.

Challenges and Learning

What was the biggest challenge you faced this week? How did you address it?

Narrowing down which techniques to use when updating the statistic. Addressed this by performing tests for each one and seeing which methods worked best with our data — I settled on using either rank transformation or sigmoid transformation.

What's one new thing you learned or skill you improved?

A few cool statistics concepts! (different ways to normalize data).

Did you attend any team meetings? Key takeaways?

No team meetings, but the team met after the Monday 11am group meet, and we divided up tasks. We split into essentially 2 groups, with me and Albert working on developing the statistic and a web app, and the rest of the team working to research parameters and develop methods for statistic calculation.

Reflection and Planning

Your progress this week?

Wrote a few PowerPoint slides to outline my research on a few statistical methods and ways we could potentially improve the validity/usefulness of our statistic. My progress this week was still predominantly research:

• (from Weekly Accomplishments): Looked for ways to update our PEI statistic and tested a few different statistical approaches to improving the output values to be easier to comprehend (e.g. rank-based transformation and sigmoid transformation). Also created notes for ways to make the statistic more useful/valid/realistic (i.e., so that the statistic is not just relative to the city it is in, but to the whole country).

Main focus for next week?

• Outline how we are going to create a barebones React web app to display data.
• Write a first piece of code that helps smooth out our statistic (implement code for rank or sigmoid transformation).
• Continue to make a plan regarding parameters and subindices to look back at when it's time to write some code!

Any resources you're looking for?

N/A

[AtharvaBeesen]

Weekly Notebook Entry — Week 3

Quick Overview

List your top 3 tasks or objectives for this week:

• Task 1: Develop an idea for our web app and create mockups in Figma.
• Task 2: Think about the logic regarding creating an API for our web app.
• Task 3: Begin writing code for the web app components, particularly the Leaflet map that will display our data.

Weekly Accomplishments

What tasks did you complete this week?

Designed the UI in Figma and wrote code (JavaScript, HTML, CSS) for key components required for the frontend. I also designed logic for how to create API requests to our backend Python code for parameter creation (the algorithms for the parameters have remained unchanged from the previous semester).

Tasks still ongoing? Current status?

Need to work with Albert to actually start bringing multiple components of the web app together. Also need to work with Chunlan to improve the visual appeal of our frontend - as it is bare bones right now.

Challenges and Learning

What was the biggest challenge you faced this week? How did you address it?

Writing pseudocode to incorporate a Leaflet map into our web app - this has been an issue I have faced in the past and so I attempted to reuse old functioning code.

What's one new thing you learned or skill you improved?

Re-learning how to build RESTful APIs.

Did you attend any team meetings? Key takeaways?

The team met after the Monday 11am group meeting, and we divided up tasks. We refined the immediate tasks that the 2 groups we created from last week should focus on. Albert and I are primarily focused on fleshing out a functioning MVP web app, and Yunlan will focus on UI development.

Reflection and Planning

Your progress this week?

• My progress this week involved writing out fragments of code that we can pull together when we finalize the structure of our web app. This involved watching multiple System Design and Frontend Development tutorials to brush up my skills.
• (from Weekly Accomplishments): Designed a frontend in Figma and wrote code (Javascript, HTML, CSS) for key components required as part of the frontend. I also designed logic for how to create API requests to our backend Python code for parameter creation (the algorithms for parameter create have remained unchanged as of now from the previous semester).

Main focus for next week?

• Pull together an MVP web app!
• Figure out API logic for the web app - this will also help 3rd parties who might want to utilize just the raw output of our algorithms.

Any resources you're looking for?

N/A

[AtharvaBeesen]

Weekly Notebook Entry — Week 4

Quick Overview

List your top 3 tasks or objectives for this week:

• Task 1: Creating an MVP web app.
• Task 2: Learning more about RESTful API development.
• Task 3: Developing future plans for incorporating the API into the MVP web app.

Weekly Accomplishments

What tasks did you complete this week?

Fully built out a web app using React (JavaScript, HTML, CSS). The web app is still currently static - I just ran a PDI algorithm externally and saved the GeoJSON output into the React app as a standalone file to use when generating the interactive map on the app.

Tasks still ongoing? Current status?

Working with Albert to build out the API. Due to multiple moving parts, this will require another week or two to build out the logic and clean up the code written last semester.

Challenges and Learning

What was the biggest challenge you faced this week? How did you address it?

Building the web app was relatively difficult and we realized it would not be feasible to incorporate an API call as part of the web app. We addressed this challenge by initially just creating a static web app and looking to develop a fleshed out API in the next 2 weeks.

What's one new thing you learned or skill you improved?

Re-learning how to build React Web Apps. Also improved in my knowledge of creating RESTful APIs, and I believe that I outlined a good plan in a meeting with Albert for how we should build our custom API.

Did you attend any team meetings? Key takeaways?

Met with Albert on Sunday to fix bugs as part of the MVP web app and split up tasks for our API development.

Reflection and Planning

Your progress this week?

• My progress this week was entirely focused on building out our web app. On Saturday, I also had an hour call with Albert to discuss ideas for the API and we drew out the logic on a shared Google Docs file so we can start writing up some code for the API.
• (from Weekly Accomplishments): Fully built out a web app using React (JavaScript, HTML, CSS). The web app is still currently static - I just ran a PDI algorithm externally and saved the GeoJSON output into the React app as a standalone file to use when generating the interactive map on the app.

Main focus for next week?

• Build out the custom API (might not be doable in a week).

Any resources you're looking for?

N/A

[AtharvaBeesen]

Weekly Notebook Entry — Week 6

Quick Overview

List your top tasks or objectives for this week:

• Task 1: GATHER HISTORICAL DATA AS FAR BACK AS POSSIBLE AND ADD THIS TO THE WEB APP - get at least one subindex up and running.
• Task 2: Try and get one of the OSM API scrapers to work and provide data, and also try and see how far we can get before the data is no longer complete/useful.

Weekly Accomplishments

What tasks did you complete this week?

Managed to finally get an API query to return usable data on 2013 data (this was the furthest back I could get). The Overpass API function managed to work for the IDI subindex by counting street "nodes" as intersections.

In the current OSM data, intersections are more clearly marked, whereas for the 2013 data I had to search for key fields such as street crossing nodes, highway intersection nodes etc.

Tasks still ongoing? Current status?

Trying to create datasets for the remaining subindexes for data between 2013 and 2022. Need to figure out how to use the Overpass API with our different subindexes.

Challenges and Learning

What was the biggest challenge you faced this week? How did you address it?

Getting the Overpass API to run was extremely difficult. To address this I essentially had to use trial and error - once I finally found a working methodology, I wrote this down as documentation for the next time I may need to use the API.

What's one new thing you learned or skill you improved?

Re-learning Python Flask and its API-based functions.

Did you attend any team meetings? Key takeaways?

Met with the team to see how they felt about static data, and how Chunlan may want us to format the data so it better aligns with her UI/UX ideas.

Main focus for next week?

• Completing the rest of the subindexes.

Any resources you're looking for?

N/A

[AtharvaBeesen]

Weekly Notebook Entry — Week 7

Quick Overview

List your top tasks or objectives for this week:

• Task 1: Continuing to build out remaining subindices for Atlanta Census block groups between 2013 and 2022.
• Task 2: Figuring out a consistent approach to gathering data using the Overpass API.

Weekly Accomplishments

What tasks did you complete this week?

Added button to web app that allows users to download the raw csv or geojson data that they are looking at! Also completed a few more of the subindices for 2013 and 2022.

Tasks still ongoing? Current status?

Finalizing datasets for the different subindices. Almost finished! Done with 3/4 of the subindeces now between 2013 and 2022.

Challenges and Learning

What was the biggest challenge you faced this week? How did you address it?

Getting the Overpass API calling structure to work. Was also encountering lots of issues regarding projecting historical data onto the map as Overpass used a different coordinate structure in 2013. I was able to figure out which system they used via an old reddit post and projected the coordinates accordingly.

What's one new thing you learned or skill you improved?

React and Javascript when updating the web app.

Did you attend any team meetings? Key takeaways?

Met with Albert on Sunday to figure out how to implement out LDI and potentiall how to optimize it - as it would often take >30 minutes to run.

Main focus for next week?

• See what Dr Ku has to say in our meeting for next week and take next steps accordingly.

Any resources you're looking for?

N/A

[AtharvaBeesen]

Weekly Notebook Entry — Week 8

Quick Overview

List your top 3 tasks or objectives for this week:

• Task 1: Create a dataset containing census tract data for the whole US for Dr Ku.
• Task 2: Getting data on a larger scale - the whole US and not just specific cities.
• Task 3: Attempting to run our model in the background as it will take a week or so to calculate all of our subindexes.

Weekly Accomplishments

What tasks did you complete this week?

Finalized a method and wrote documentation to pull data from Overpass API! So we should be set to try and automate data collection and hence automate subindex generation.

Tasks still ongoing? Current status?

Working with Albert to actually write code that will allow us to develop a nation wide dataset for Dr Ku.

Challenges and Learning

What was the biggest challenge you faced this week? How did you address it?

Trying to figure out how to optimize our algorithms before we run them on huge datasets. It was also tricky to try and gather data for census tracts (as we usually use census block groups).

What's one new thing you learned or skill you improved?

Custom API building and testing within Python.

Did you attend any team meetings? Key takeaways?

Met with Dr Ku on Monday to discuss his goals and what he wants from us. We learnt that he was less interested in the Web App and more interested in a nation wide dataset of our parameters.

Main focus for next week?

• Build out the dataset for Dr Ku.

Any resources you're looking for?

N/A