311 Data Weekly Engineering Meeting

ryanfchase commented 6 months ago

Overview

Meeting agendas for the weekly 311 Data Engineering member meetings, useful links and other resources

Agenda Quick Links

2024-05-08

Resources/Instructions

Engineering Onboard/Offboard Issue

Template for weekly meeting's comment.

## YYYY-MM-DD - 311 Data Weekly Engineering Meeting
_[Back to Top](#)_

### Quick Links
- [Open PRs](https://github.com/hackforla/311-data/pulls)
- [Ready For Dev Lead](https://github.com/hackforla/311-data/issues?q=is%3Aissue+is%3Aopen+label%3A%22ready+for+dev+lead%22+-label%3Adraft+label%3A%22Role%3A+Frontend%22)
- [Bug Hunt Live-Forever Ticket](https://github.com/hackforla/311-data/issues/1726)
- [Frontend Project Board](https://github.com/orgs/hackforla/projects/63/views/1?filterQuery=label%3A%22Role%3A+Frontend%22)
  - [Issues Ready To Be Worked On](https://github.com/orgs/hackforla/projects/63/views/1?filterQuery=label%3A%22Role%3A+Frontend%22+no%3Aassignee+status%3A%22Prioritized+Backlog%22)
  - [What's Being Worked On (Frontend)](https://github.com/orgs/hackforla/projects/63/views/1?filterQuery=has%3Aassignee+label%3A%22Role%3A+Frontend%22)
  - [Developer Self Test](https://docs.google.com/presentation/d/1RUQk4mOrv4G1fja_D3GMWMtCyCqsdj64fNPlrIDJhxg/edit#slide=id.g114a9db5138_0_0)
- [Github Pages Dev Site](https://hackforla.github.io/311-data/#/map)
- [Engineering Onboarding Issue](https://github.com/hackforla/311-data/issues/1578)
- [311 Roster](https://docs.google.com/spreadsheets/d/1OsLDl7Ciwj7WjtzbgDz38g4kpOMNlUhdjlgzQxtQTvM/edit#gid=0)

### Topic requests/questions:

### Action Items:

### Resources / Links:

### Attendees

- [ ] Tracy
- [ ] Ryan C
- [ ] Shannan
- [ ] Billy Phan
- [ ] Danielle
- [ ] Kelly
- [ ] Andrew
- [ ] Brandon
- [ ] Dorian
- [ ] Mrudula

ryanfchase commented 6 months ago

2024-05-08 - 311 Data Weekly Engineering Meeting

Back to Top

Weekly Announcement Volunteer

@Skydodle

Quick Links

Topic requests/questions:

[x] Weekly Engineering agenda
[x] Dev team announcements person
[x] Create open role on Engineering CoP
[x] Bug-hunt ticket
[x] Contact Form
[x] Discuss new ticket goals

Action Items

[ ] brainstorm documentation issues (3 or 4 updates / new pages)

Resources / Links:

More info on today's Topic Requests

``` 1. Weekly Engineering agenda - we can briefly list all open tickets here, list PRs here, list goals here 2. Dev team announcements person: - designated member to announce PRs, open tickets, etc (weekly rotation) - assign this person at the weekly meeting 3. Our plan to get new members: - create an Open Role on Engineering CoP - to justify this, we should make sure we have available tickets, and good onboarding + documentation 4. Bug-hunt & Code Quality tickets - a live forever ticket that gets reassigned every week, challenge to break the code - person assigned can do a bug writeup - can be a good first issue - Can document proposed solutions to these bugs and prioritize them for later 5. Contact Form - Create a Github access token for 311 Data dev 6. New Ticket Goals - try and maintain a number of Good First Issues (e.g. 3-5) - The number of available tickets should roughly match number of active devs ```

Attendees

[ ] Ryan C
[ ] Johnny
[ ] Shannan
[ ] Billy Phan
[ ] Drake

ryanfchase commented 6 months ago

2024-05-18 - 311 Data Weekly Engineering Meeting

Back to Top

Next Week's Announcement Volunteer

@ryanfchase

Quick Links

Topic requests/questions:

[x] live forever bug reporting ticket

Action Items:

Resources / Links:

Attendees

[ ] Ryan C
[x] Johnny
[ ] Shannan
[ ] Billy Phan
[ ] Drake

ryanfchase commented 6 months ago

2024-05-22 - 311 Data Weekly Engineering Meeting

Back to Top

Next Week's Announcement Volunteer

Quick Links

Topic requests/questions:

Action Items:

review Engineering Documentation
- CONTRIBUTING.md
- README.md

Resources / Links:

Attendees

[ ] Ryan C
[ ] Johnny
[ ] Shannan
[ ] Billy Phan
[ ] Drake

ryanfchase commented 6 months ago

Posting our meeting notes from Wednesday...

Notes on Readme

311-Data v2 --> 311-Data v1.2
update the screenshot of the app
remove Matplotlib, Polars, Numpy
remove Google Colab
remove Observable (tbd)

Notes on contributing.md

if you run into issues, post onto #311-data-engineering, (not open new issue -- we dont have infrastructure for this)
feature branching format: [ issueId-Title, issueId-MinimalDescription ]

TODO

prune branches on 311-Data
see if we want to implement any linting rules, PR rules for github actions
Getting started, it would be nice if there was a directory tree with main components, and some descriptions of what they do

ryanfchase commented 5 months ago

2024-06-12- 311 Data Weekly Engineering Meeting

Back to Top

Next Week's Announcement Volunteer

(replace with @ github handle)

Quick Links

Self-Intros

Returning members (1-3 minutes)
1. If we have new members present, brief introduction about yourself.
2. What have you completed since last Tuesday?
3. What do you plan to complete before next Tuesday?
4. If you have a blocker, leave a short summary in the chat so we can help you resolve it later in the meeting
New members (1-3 minutes)
1. Where are you from?
2. What are you looking to get out of volunteering with Hack for LA?
3. What is your experience with coding / open source contribution?

Topic requests/questions:

Review Assigned Tickets

[x] Johnny
[x] Kelly
[x] Andrew
[x] Mrudula
[x] Jiazong
[x] Anna

Questions from new members:

[x] What is the primary purpose of the application
[x] What's our roadmap for Dashboard related features
[x] What documentation do we have for high-level data flow
[x] What's our plan for frontend testing coverage
[x] Where are user flows documented (fwd from Andrew's comment here). Answer: Figma Page. Do we need to update a base level of documentation for this?

RC, review with team

[ ] is there more work to be done based on this ticket? https://github.com/hackforla/311-data/issues/1535#issuecomment-2147885974
[ ] efforts to organize the codebase: https://github.com/hackforla/311-data/issues/1587#issuecomment-2147871736
[ ] is there a problem with these 2 NCs? https://github.com/hackforla/311-data/issues/1540#issue-1843763181
[ ] Turn notes from 2 wk ago meeting into dev issues: https://github.com/hackforla/311-data/issues/1720#issuecomment-2130029070
[x] Review new bug-report: https://github.com/hackforla/311-data/issues/1726#issuecomment-2163764526
[x] Review with @aqandrew the frontend testing ticket: https://github.com/hackforla/311-data/issues/1334#issuecomment-2159161532

Action Items:

[x] onboard Kelly
[x] onboard Andrew
[x] onboard Danielle

RC

[ ] Review website team wiki and see if we should adopt / reference it. (note: it will change soon!)
[ ] The header should still be clickable while the loading modal is active
[ ] First documentation ticket for high level data flow, high level tech stack
- [ ] consider Excalidraw
[X] create epic for Implementing Frontend Tests: https://github.com/hackforla/311-data/issues/1768
[ ] modify Eng Onboarding to add explicit link to quickstart
- [ ] see https://github.com/hackforla/311-data/issues/1578#issuecomment-2159550797
[ ] Add limitations for pre-2021 data to a page on the wiki: https://github.com/hackforla/311-data/issues/1714#issuecomment-2161312443

I've moved these action items to the PM team agenda for tracking / updating PM issues

[ ] 311-Data bot by next week (OR at Bonnie 1 on 1 interview)

[ ] PMs come up with a human name (bot names might get flagged by github)

[ ] Admin makes a hackforla email address

[ ] PM team adds credentials to 1Password

[ ] PM team creates github account

[ ] Bonnie/Admin adds bot's github to HackForLA org

[ ] PMs to add team permissions to this bot

[ ] finish automation implementation, as outlined in this dev ticket

[ ] Scheduling a meeting with Bonnie for the Story of 311 Data page

[ ] use the Bonnie scheduling sheet to pick a time

[ ] pick one PM to conduct the interview

Resources / Links:

Attendees

[x] Ryan C
[ ] Johnny
[x] Shannan
[ ] Billy Phan
[ ] Drake
[x] Kelly
[x] Andrew
[x] Danielle

ryanfchase commented 5 months ago

2024-06-19 - 311 Data Weekly Engineering Meeting

Back to Top

Next Week's Announcement Volunteer

(replace with @ github handle)

Quick Links

Topic requests/questions:

[ ] What to do about converting the dreaded {' '} to  
- [ ] see Johnny's comment: https://github.com/hackforla/311-data/pull/1761#issuecomment-2169065261
[ ] Review rebasing: https://github.com/hackforla/311-data/pull/1766
- [ ] How do some PRs require a merge commit from your own branch?
- [ ] How do some PRs include new commits from main as part of the Files Changed?
[ ] Review some tech stack research items
- [ ] Wiki page on storing data on the repo
- [ ] Repo with data stored locally (used for comparing load times with base repo):
  Action Items:

Resources / Links:

Attendees

[ ] Ryan C
[ ] Johnny
[ ] Shannan
[ ] Billy Phan
[ ] Drake
[ ] Danielle
[ ] Kelly
[x] Andrew

ryanfchase commented 4 months ago

Leftover Action Items for @ryanfchase

[ ] Review website team wiki and see if we should adopt / reference it. (note: it will change soon!)
- [ ] review CONTRIBUTING.md (or equivalent)
[x] The header should still be clickable while the loading modal is active: https://github.com/hackforla/311-data/issues/1783
[ ] First documentation ticket for high level data flow, high level tech stack
- [ ] consider Excalidraw
[ ] modify Eng Onboarding to add explicit link to quickstart
[x] Update Readme.md: https://github.com/hackforla/311-data/issues/1784
[x] Updated export button ticket for dev: https://github.com/hackforla/311-data/issues/1695
- [x] New ticket: https://github.com/hackforla/311-data/issues/1785
[x] Complete data loading ticket: https://github.com/hackforla/311-data/issues/1716
[ ] Add to bugfix (see screenshot)

Bugfix needed for popup being covered by address bar

![image](https://github.com/hackforla/311-data/assets/6414668/23a4eff2-df88-43bd-8679-ca6288381596)

ryanfchase commented 4 months ago

2024-06-26 - 311 Data Weekly Engineering Meeting

Back to Top

Next Week's Announcement Volunteer

(replace with @ github handle)

Quick Links

Accountability Checks

Details

- [ ] [Andrew](https://github.com/hackforla/311-data/issues/1334) - [ ] [Johnny](https://github.com/hackforla/311-data/issues/1780)

Topic requests/questions:

[x] Accountability Checks: https://github.com/hackforla/311-data/issues/1545#issuecomment-2190221986
[x] review open PRs
[ ] see my comment above for unchecked boxes (need help creating tickets / writing bug reports)
[x] discussion on Vite and Testing
- [x] Vite: https://github.com/hackforla/311-data/issues/1778
- [ ] Testing Epic: https://github.com/hackforla/311-data/issues/1768
[x] address: https://github.com/hackforla/311-data/pull/1752#pullrequestreview-2142794357

Action Items:

[ ] onboard Palak
[ ] onboard Terrance

Resources / Links:

Attendees

[ ] Ryan C
[ ] Johnny
[ ] Shannan
[ ] Billy Phan
[ ] Drake
[ ] Danielle
[ ] Kelly
[x] Andrew

ryanfchase commented 4 months ago

RC TODO:

Add dependency to this: https://github.com/hackforla/311-data/issues/1334
- blocked by: https://github.com/hackforla/311-data/issues/1778
Modify boundaries from 2019 to 2023(24?) boundaries
- inspect new boundary for "dead zones"
- identify "dead zones" (aka areas that are not parks/lakes, etc and they have SR in them)
Zoom Controls: https://github.com/hackforla/311-data/issues/1700

ryanfchase commented 3 months ago

2024-08-24 - 311 Data Weekly Engineering Meeting

Back to Top

Next Week's Announcement Volunteer

(replace with @ github handle)

Quick Links

Topic requests/questions:

[ ] Complete Onboarding
- [ ] Brandon Yu
- [ ] Dorian Deptuch

Documentation Items

[ ] Frontend
- [x] Mapbox (Billy)
- [ ] React
- [ ] MUI (Shannon?)
[ ] Backend
- [ ] DuckDb
- [ ] HuggingFace
- [ ] python data transform + Github Actions
- [ ] DbProvider + Context
- [ ] Data Export Queries
- [x] Bot Email + Support Ticket Automation (Robleh)

Action Items:

ryanfchase commented 2 months ago

Meeting Notes from 8/31/24

users thought calendar would expand after clicking the date
- allow date dropdown to appear when clicking on date
- clicking on "start date" should open month that contains "start date"
- clicking on "end date" should open month that contains "end date"
- OR clicking on anywhere on the text area emulates behavior of clicking on Calendar box
users want to be able to enter the date
- this may require much more work from design+dev to cover all "error states"
- might be simpler to move forward with clickable textbox, then consider typed-out-dates

ryanfchase commented 2 months ago

2024-09-07- 311 Data Weekly Engineering Meeting

Back to Top

Quick Links

Topic requests/questions:

[x] Tracy stepping into new dev lead role

Action Items:

Resources / Links:

Attendees

[x] Ryan C
[ ] Johnny
[x] Shannan
[ ] Billy Phan
[ ] Drake
[ ] Danielle
[ ] Kelly
[ ] Andrew
[x] Brandon
[x] Dorian
[x] Tracy

ryanfchase commented 2 months ago

Borrowing from Product Planning new tickets comment.

New tickets checklist for eng:

[X] Add NC name pill to Search and Filters modal on address selection: https://github.com/hackforla/311-data/issues/1813
[X] Clicking on date range should drop down calendar: https://github.com/hackforla/311-data/issues/1806
[X] Fix SR detail loading spinner: https://github.com/hackforla/311-data/issues/1805

Blocking ticket for #1708:

[x] obtain latest NC boundary data: https://github.com/hackforla/311-data/issues/1814

Other tickets

[x] Make font types in search and filters modal consistent: https://github.com/hackforla/311-data/issues/1815

ryanfchase commented 2 months ago

2024-09-14 - 311 Data Weekly Engineering Meeting

Back to Top

AGENDA

Timezones listed as Pacific Time.

[x] 10:00AM 5 min - Check-in
[ ] 10:05AM 5 min - Team announcements + updates
[ ] 10:10AM 20 min - Topic Requests
[ ] 10:30AM 30 min - Breakout Rooms
[ ] 11:00AM finished

Topic requests/questions:

RC

[x] Tracy Intro!!!
[x] Pair Programming
- [x] https://github.com/hackforla/311-data/issues/1720#issuecomment-2351065978
[X] what's the purpose of dispatchUpdateUnselectedCouncils ?
[x] https://github.com/hackforla/311-data/issues/1820
[ ] review Andrew's comments on https://github.com/hackforla/311-data/issues/1778
[ ] maybe cover some highlights of https://github.com/hackforla/311-data/pull/1822

Action Items:

Resources / Links:

Attendees

[x] Ryan C
[ ] Johnny
[x] Shannan
[ ] Billy Phan
[ ] Drake
[x] Danielle
[x] Kelly
[ ] Andrew
[x] Brandon
[ ] Dorian
[x] Tracy
[ ] Mrudula

ryanfchase commented 2 months ago

2024-09-14 Notes

Pair programming
- rubber ducky discussion
- co-working zoom and/or slack --> set one up for all of HackForLA
- can we pilot this in #311-data-engineering ?
- we'd need a way to set up a meeting link that doesn't expire / doesn't rely on host to be there
- Slack huddles doesn't work because we have a Free account. It currently only supports 2 people in a huddle.
- HackForLA discord server? easier to do co-working groups
unselected / updateUnselectedCouncils
- this makes sense if we were trying to propagate a history of unselected councils
- but sometimes we end up using the entire councils prop (see L270 in Map.jsx)
- we should see if unselected used to be used for something useful, otherwise try to remove it
make this ticket an epic: https://github.com/hackforla/311-data/issues/1820
- consider doing a few major components first as an example
- then start provisioning different issues to tackle groups of making this change
- consider good first issue
- consider good pair programming issues
- TODO: make this a label
discussed project board automations
- GraphQL query for obtaining board metadata: https://docs.github.com/en/enterprise-server@3.12/graphql/overview/explorer
discussed Storybook
- https://storybook.js.org/tutorials/intro-to-storybook/react/en/get-started/

ryanfchase commented 2 months ago

2024-09-21 - 311 Data Weekly Engineering Meeting

Back to Top

Next Week's Announcement Volunteer

(replace with @ github handle)

Quick Links

Topic requests/questions:

[ ] Discuss Vite
- [ ] consider merging this after "launch"
- [ ] consider recreating our build process, compare to new vite process (use a simplified example)

Action Items:

Resources / Links:

Attendees

[ ] Tracy
[ ] Ryan C
[ ] Shannan
[ ] Billy Phan
[ ] Danielle
[ ] Kelly
[ ] Andrew
[ ] Brandon
[ ] Dorian
[ ] Mrudula

ryanfchase commented 1 month ago

2024-09-28 - 311 Data Weekly Engineering Meeting

Back to Top

Quick Links

Topic requests/questions:

[X] Review Wednesday meeting's agenda
- [X] discussed Contact Us page new tx upcoming (see R2.1)
[X] Leetcode
- [X] Palindrome number (see R1.1, R1.2)
- [ ] For next time: Combination Sum

Action Items:

Resources / Links:

Links used in Leetcode 1.1 How many bytes in a javascript string 1.2 How many bytes in a javascript integer
Discussing Contact Page 2.1 What is PII

Attendees

[ ] Tracy
[ ] Ryan C
[ ] Shannan
[ ] Billy Phan
[ ] Danielle
[ ] Kelly
[ ] Andrew
[ ] Brandon
[ ] Dorian
[ ] Mrudula

traycn commented 1 month ago

2024-10-05 - 311 Data Weekly Engineering Meeting

Back to Top

Quick Links

Topic requests/questions:

Action Items:

Resources / Links:

Attendees

[x] Tracy
[ ] Ryan C
[ ] Shannan
[ ] Billy Phan
[ ] Danielle
[ ] Kelly
[ ] Andrew
[ ] Brandon
[x] Dorian
[x] Mrudula

ryanfchase commented 1 month ago

For next agenda

Need to discuss Vite build issue from VRMS (see if there is any overlap with our project)

https://github.com/hackforla/VRMS/issues/1767

ryanfchase commented 1 month ago

2024-10-12 - 311 Data Weekly Engineering Meeting

Back to Top

Quick Links

Topic requests/questions:

Action Items:

Resources / Links:

Attendees

[ ] Tracy
[ ] Ryan C
[ ] Shannan
[ ] Billy Phan
[ ] Danielle
[ ] Kelly
[ ] Andrew
[ ] Brandon
[ ] Dorian
[ ] Mrudula

ryanfchase commented 1 month ago

2024-10-12 Extended Session (WIP)

Notes

In our meeting we covered...

https://github.com/hackforla/311-data/issues/1716
- these metrics are useful, but ultimately they tell us what we already know: the browser is running out of memory for 3 months of data and onward
- could consider closing this ticket
an older ticket regarding splitting up the data, ultimately the correct direction to consider: https://github.com/hackforla/311-data/issues/1542
why are we making so many event api calls? Tilesets makes sense, since you need to grab the tileset before you render it. But why events?
- control api calls via transformRequest? https://stackoverflow.com/questions/61221421/mapbox-gl-js-api-calls-control
@DrAcula27 noticed that we are making API calls to the terrain styles API... perhaps we need to turn this off
- https://api.mapbox.com/v4/mapbox.mapbox-streets-v8,mapbox.mapbox-terrain-v2,mapbox.mapbox-bathymetry-v2/13/1402/3271.vector.pbf?style=mapbox://styles/mapbox/light-v11@0&sku=101dzVAySrFTJ&access_token=<redacted>
@traycn provided a resource to an optimization guide for mapbox: https://docs.mapbox.com/help/troubleshooting/mapbox-gl-js-performance/

Performance Profiling

@bphan002 and @ryanfchase attempted to use the Chrome DevTools profiler to record how much memory our application was incurring.
- Ryan Profile 1: profile crashed, reported running out of memory, when rendering 3 months of data w/ all request types (8GB RAM, Chrome, while screen sharing)
- Ryan Profile 2: profile experienced some slowdowns, but did not crash, rendering 3 months of data w/ 1 request types (same conditions)
- Billy Profile 1: profile ran fine while rendering 3 months of data w/ 1 request type(16GB RAM, Chrome, needs confirmation)
- Billy Profile 2: profile ran fine whiel rendering 3 months of data w/ 2 request types, however the application failed to respond when attempting a 3rd request type (bulky items, electronic waste, homeless encampment?)

After the meeting, continuing to look into profilers...

Ryan Profile 3: 3 months of data, one SR type at a time, up to 3 SRs
- I was seeing 2GB - 2.5GB of heap size for the JS VM
- Task Manager reported Chrome at 80~85% Memory usage
- I could identify which Chrome sub-process was requiring 2 - 3GB of RAM
- site did not crash, but I did experience some stutters and skipped inputs (also my music, playing in youtube, stopped for a brief moment)
- I eventually enabled ALL SRs, application did not immediately crash.
- it wasn't until I started zooming in/out on the map that I received this exception (see dropdown)
- This exception doesn't necessarily indicate that we ran out of memory. Running out of memory looks different (see dropdown)
- this.expression.evaluate is a method used while mapbox is determining how to render styles and layout of its data. I noticed that while I was profiling, the boundary fills stopped rendering, as if mapbox forgot to do it / skipped it. One possibility is that mapbox attempted to execute an expression but some resource or object was missing. Perhaps we're seeing a memory leak (probably more precisely, a memory management problem)

Profile 3: exception and stack trace

![image](https://github.com/user-attachments/assets/7450ec82-298e-4404-964b-629136d414fe)

Out of memory icon

![image](https://github.com/user-attachments/assets/d1fadf41-ecbd-4b4d-ab4d-f6d88fe7ebc1)

Memory Profiler: Looking at Memory Snapshots rendering 3 months of data w/ ALL service request types:

NOTE: I'm actually not sure if I'm interpreting these tables correctly. Will need to continue looking into them.

Object: Possible duplication of memory

These are listed under `Object`, I assume that means these are just JS objects living in memory. All other entries in thist list were 2MB or less SR Requests being stored in React: ~45MB ![image](https://github.com/user-attachments/assets/5b618642-3f0e-4205-af52-ce44148854fd) SR Requests being stored in Mapbox: ~45MB ![image](https://github.com/user-attachments/assets/25819764-259b-4408-93db-a87f063b48bc) Top 6 Biggest `Object` data structures ![image](https://github.com/user-attachments/assets/b0570a1e-7188-4474-a6cd-f539f79fd019)

MapContainer

MapContainer: ~366MB, not sure if there's much to do here. Unsure of how to slim this, or if it needs to be slimmed. ![image](https://github.com/user-attachments/assets/fe5217a0-3bd9-4982-b369-330c8d025956)

Action Items

[ ] How much data gets loaded on page-load?
- pretty sure we just go to hugging face and look at how much data is in the 2024 data set: ~1.12MB for 2024
- But is this what's causing the application to crash?
- Guess 1: No.
  - It takes a long time to load the data and render the SRs, but this does not inherently crash the site
  - site will crash as soon as you start zooming in and out
  - we observe that there are MANY api calls to mapbox that occur as you zoom in/out of the map. Some are fetching tilesets, some are api calls to events

Resources

Mapbox Performance Guide: https://docs.mapbox.com/help/troubleshooting/mapbox-gl-js-performance/
Chrome DevTools Profiler: https://developer.chrome.com/docs/devtools/performance
React Profiler: https://legacy.reactjs.org/blog/2018/09/10/introducing-the-react-profiler.html
Clearing Mapbox in-memory storage: https://github.com/mapbox/mapbox-gl-js/issues/9126
Huggingface Repo: https://huggingface.co/311-data

ryanfchase commented 1 month ago

2024-10-19 - 311 Data Weekly Engineering Meeting

Back to Top

Quick Links

Topic requests/questions:

Action Items:

Resources / Links:

Attendees

[ ] Tracy
[x] Ryan C
[ ] Shannan
[x] Billy Phan
[x] Danielle
[x] Kelly
[ ] Andrew
[ ] Brandon
[ ] Dorian
[x] Mrudula

ryanfchase commented 1 month ago

Notes from 10/19

talked about maybe limiting RequestLayer's props, we are doing a comparison of requests against prev.requests, and this is potentially a really really bad performance action
new ticket: https://github.com/hackforla/311-data/issues/1846
Pre-work checklist: https://github.com/hackforla/website/issues/6598

ryanfchase commented 1 month ago

Performance Notes (Cont)

I decided to try and see exactly how we were pulling 2023 data (since this isn't pulled on page-load). I noticed that it was actually only capable of pulling data from hugging face in 16.8MB sized chunks. See picture. This suggests that if we are to split up the data into manageable chunks, that 16.8MB is sort of the high end of resource size.

I noticed there was 4.2MB in the first chunk, proceeded by seven 16.8MB chunks, and then 8.3MB in the last chunk. That's about 130MB of data. And if you go to our Huggingface repo, you'll see that the 2023.parquet is 135MB large, so that's pretty close to what I expected.

2023 parquet being organized into chunks

![image](https://github.com/user-attachments/assets/a4340282-b0a9-40fc-b7e7-2569d9be2b6f)

ryanfchase commented 1 week ago

2024-11-09 - 311 Data Weekly Engineering Meeting

Back to Top

Quick Links

Topic requests/questions:

[x] 311 Ryan
- [x] Recreating our map
- [x] Recreating our data fetching system
[ ] community resource sharing
Action Items:

Resources / Links:

Attendees

[x] Tracy
[x] Ryan C
[ ] Shannan
[x] Billy Phan
[ ] Danielle
[x] Kelly
[ ] Andrew
[ ] Brandon
[ ] Dorian
[ ] Mrudula

hackforla / 311-data