search

NYCPlanning / data-engineering-qaqc

streamlit app for data engineering
https://edm-data-engineering.nycplanningdigital.com
1 stars 0 forks source link

Draft: 182: Caching and Digital Ocean Abstraction #198

Closed abrieff closed 2 years ago

abrieff commented 2 years ago

See https://github.com/NYCPlanning/data-engineering-qaqc/pull/206

abrieff commented 2 years ago

Overall this refactor looks great and will DRY out a ton of code. My only skepticism is with the caching. I really like that you took initiative to implement this complex good feature, but I'm not convinced we are good enough programmers to maintain it. I'm not sure it's worth the effort for DE to learn and maintain a feature that we probably won't use anywhere else and doesn't really tie in with our other programming. Some of the code Baiyue wrote that none of us really understand has worked fine for months and months so maybe it will be like that. But I predict that this app's maintenance will already be a lot of work without any caching, seeing that it was written when we had about twice as many engineering hours per week as we will going forward. I think we should all get on a call in the next couple days and we can talk about this more. Does that seem like a good next step to you?

This seems basically right/reasonable - I can seperate the two things out, write up my thoughts about the caching in an issue, and leave it for you all if it comes up again in the future.

SashaWeinstein commented 2 years ago

Ok sick, glad that seems reasonable. Does it make sense for all five of us to get on a call to talk about caching? If Te and Max think it's maintainable then maybe it does make sense to push it to dev

abrieff commented 2 years ago

Sure - maybe we can make time before or after standup

abrieff commented 2 years ago

Closing this because i've opened a new PR with just the caching.