StylusEater
commented
3 years ago name: Crates.io Weekly Team Meeting about: Create a new weekly meeting agenda. title: 'Crates.io Weekly Team Meeting #XX' labels: meeting assignees: ''
Meeting Details
- Date/Time: Friday, September 11, 2020, 2030 UTC
- 1430 UTC
- Location: Link will be posted in #crates-io-team channel on Discord shortly before the call.
- Recording: TBA
- Moderator: TBA
Proposed Agenda
- Need a volunteer to take notes
- Confirm if we will switch to new meeting time (16:00 UTC, 1.5 hours later than current time) starting next week
- Roadmap
- eRFC: Crate name transfer
Roadmap v0.1
Roadmap v0.1
- Growing the team
- The biggest area I think we can use help at the moment is with PR reviews and issue triage. We don’t have a huge backlog of PRs, but if something isn’t acted on right away, it can easily go weeks or months without a review. It’s hard to keep people involved if their contributions just sit there with no apparent engagement. (I don’t mean to make things sound dire. This isn’t every PR, or even most, but it does happen. I think we’re just stuck in a moment where we may be missing opportunities to pull people in because we don’t have the existing bandwidth to be great at it.)
- Security and ops improvements
- Some of this work is reactive and preemptive. An outage or security issue is immediately an urgent priority.
- Lots of incremental work to improve architecture, stay current on dependencies (also see below), and build defense in depth.
- We should have a team area to manage important ops information. The ops channel works pretty well, but isn't great for info we need to curate long-term. We should capture this knowledge and earned experience to help with on-boarding new members.
- The codebase (by definition) is older than the ecosystem, and we should continue to leverage the ecosystem where practical. Long term, we shouldn’t be maintaining our own logic for secure cookies, serving static assets, setting sane security response headers, etc.
- Support channels: how do we efficiently and effectively manage the growing support request workload? (Maybe this should have its own top-level item.)
- Design updates and polish of existing features
- The design refresh ties in with the web-presence team and this work can largely progress independently of the backend and infrastructure.
- Independent of the redesign, I think there are various paper-cuts that can be addressed and hopefully these types of changes are small and obvious wins that can get through review easily.
- This isn't just for the frontend. We've had requests for API improvements, but there are tradeoffs:
- Some consumers want less data. In some cases, I think the frontend gets data in a different request but we continue to send data on the original endpoint because a 3rd party consumer could exist. In other cases, the payload makes sense but there are edge cases (like a crate that has many features) that cause the payload to grow to 1MB+ when a particular consumer only wants a few small pieces of info.
- Some consumers want more data. Do we add to an existing endpoint or create a new one? We offer database dumps, but some consumers (like browser extensions) can't use the dumps or the registry index in practice.
- We should start developing a process for deprecating v1 endpoints as the need arises. Even if we never plan to remove the endpoint, we should document that new consumers would be better off starting a conversation with us about their needs. We should have a way to declare endpoints as an internal interface (for usage by frontend or ops only). For example, the dashboard and email notifications are fairly specific to our UI. When we add new API, it should be unstable by default unless we explicitly decide to make some or all fields stable.
- We should find a way to engage with 3rd party consumers so that we have more insight into their usage of the API. For example, it would be more efficient for us to offer an API to query all newly published versions (possibly since the previous timestamp or ID number) than to crawl through pages of search results.
- Refactoring, keeping deps up to date, and developer experience.
- Technical debt
- We’ve made a lot of progress on technical debt over the last few years, there is still room to improve. This falls under developer experience too, making the codebase more approachable.
- In the controller logic, we can run into some really tough lifetime situations with the request extensions. For instance, you can’t have a database connection in scope and also mutate the session cookie, because of lifetime issues on the &mut dyn RequestExt. (JTG: I have an idea for a change that I hope fixes the root-cause, but I’m not sure yet.)
- Maintenance tasks
- For frontend deps various bots have been working well.
- For backend we can now run cargo update safely and investigate bots for this yet. Doing bulk updates (like I think dependabot would) causes the next deploy to take a long time and grows the build cache on Heroku if done frequently.
- We currently pin to a specific version of rustc for production, and try to bump after a new stable release. If we had a rough schedule of when we bump dependencies and rustc, then an easy way for someone to contribute would be to open the appropriate PRs every 2 or 3 weeks.
- Developer experience - that is, working with the crates.io codebase. (User experience through cargo or the website fall under polish above I think.)
- Compile times are worse than necessary because we have several one-off binaries for various ops things (many that I’ve never even personally run, are they all still necessary?). It would be great to consolidate most of these into just a few binaries.
- Running cargo test without -j 8 OOMs Pietro’s workstation (16GB of RAM) when trying to link all those binaries in parallel.
- Improvements to documentation
- Improvements to getting a local setup running, with a background worker, job scheduling, etc.
Outcomes
-
Choosing a new Framework SUGGESTION: Instead of using a custom framework then maybe go to a standardized framework? NOTE: We did take over the Conduit crates so maybe there is something we can help with there? ** CONCERN: We have production code working so it’s very risky to go to something new.
-
Growing the Team ** Consensus is we need to grow the team.
-
Security and Ops Improvements Infrastructure stability is super important NOTE: We need more defense in depth NEED: A place to put private documentation for the operations team we already have an infra GitHub repository (maybe this is a good place?) should we find something beyond GitHub and issue tracking this could potentially tie into the support channel E-mail management tool? Have a trial account with HelpMonks but might not have OAuth with GitHub Have a Beta for Yetto but haven’t tried Explore Mailgun and GitHub to make private issues We need a nice interface to transfer crates and verify crate owners some people in Crates.io have been adding notes people are willing to give up crate names Web interface would be great to have … easy transfer, delete, add … docsrus, S3, crates.io (especially DMCA requests) … it would be nice to have an auto-reply support requests at NPM were more from publish to unpublished because of … THERE ARE MANY STANDARD ITEMS Delete process is partially done Need more automation in Security and Ops … Ops Automation Enable people to delete their own crate within 24hrs of publishing ** TODO: Need to figure out how to better handle large bursts of download requests … maybe generate a cache to buffer requests?
-
Discuss eRFC https://internals.rust-lang.org/t/pre-erfc-crate-name-transfer/9027 No resolution on this today ** SUGGESTION: Outline blockers, make things clear that other aspects need to be considered and implemented before we can do what is being requested. (Blockers for the eRFC: https://github.com/dtolnay/rfcs/blob/transfer/text/0000-crate-name-transfer.md#blockers)
-
Name Spacing and Name Squatting SUGGESTION: Name Squatting … have an approval process for name squatting Talk to people from NPM so we try to avoid pitfalls they encountered
-
Zoom Meeting ** Suggestion we do a once a month (first meeting of the month) Zoom meeting and then the others are Discord meetings
NEXT WEEK MEETING: ** Try to nail down the next zoom meeting through Discord
TODO:
- create and propose a meeting structure template optimized for note taking
- create and propose a meeting notes structure
- create documentation on how new developers can setup their environments and develop
Action Items
- Try to nail down the next zoom meeting through Discord
- Suggestion we do a once a month (first meeting of the month) Zoom meeting and then the others are Discord meetings
jtgeibel
Here is a rough template for meetings that we might want to refactor and use going forward: