open-sauced / pizza

This is an engine that sources git commits and turns them to insights
Apache License 2.0
31 stars 13 forks source link

feat: Normalizes and validates git URLs before accepting for processing #38

Closed jpmcb closed 1 year ago

jpmcb commented 1 year ago

Description

This patch is twofold:

  1. Incoming request URLs to the bake route will be normalized. This way, we prevent duplicate entries from dropping in (like with https://github.com/open-sauced/pizza/ and https://github.com/open-sauced/pizza
  2. Validates that the repo URL is actually a git repo. This is accomplished by essentially calling git ls-remote on the provided URL which is more efficient than doing a shallow clone.

I also cleaned up where repoURL was getting called in the processRepository function

Example:

On calling my local repo with:

❯ pizza bake -e http://localhost:8080 https://github.com/jpmcb/dotfiles/

it works and correctly validates/normalizes the url. Here are the debug logs from the server 👍🏼

❯ go run main.go --debug
2023-08-29T14:35:26.033-0600    INFO    pizza/main.go:41        initiated zap logger with level: -1
2023-08-29T14:35:26.091-0600    INFO    pizza/main.go:91        Initiating cache git provider
2023-08-29T14:35:26.091-0600    INFO    server/server.go:52     Starting server on port 8080
2023-08-29T14:35:28.328-0600    DEBUG   server/server.go:78     Validating and normalizing repository URL: https://github.com/jpmcb/dotfiles/
2023-08-29T14:35:28.623-0600    DEBUG   server/server.go:144    Checking if repository is already in database: https://github.com/jpmcb/dotfiles
2023-08-29T14:35:28.643-0600    DEBUG   server/server.go:158    Getting repo via configured git provider: https://github.com/jpmcb/dotfiles
2023-08-29T14:35:28.644-0600    DEBUG   providers/cache.go:47   Getting repo from LRU cache: https://github.com/jpmcb/dotfiles
2023-08-29T14:35:28.644-0600    DEBUG   providers/cache.go:51   Cache miss. Putting to cache: https://github.com/jpmcb/dotfiles
2023-08-29T14:35:28.644-0600    DEBUG   providers/cache.go:58   Opening and fetching repo: https://github.com/jpmcb/dotfiles
2023-08-29T14:35:28.749-0600    DEBUG   server/server.go:169    Inspecting the head of the git repo: https://github.com/jpmcb/dotfiles
2023-08-29T14:35:28.749-0600    DEBUG   server/server.go:175    Getting last commit in DB: https://github.com/jpmcb/dotfiles
2023-08-29T14:35:28.754-0600    DEBUG   server/server.go:181    Querying commits since: 2022-12-08 22:47:01 +0000 UTC
2023-08-29T14:35:28.754-0600    DEBUG   server/server.go:189    Getting commit iterator with git log options: {1f7099ac02c52ff4b775ac894b2e26ed5a56207e 0 <nil> <nil> false 2022-12-08 22:47:01 +0000 UTC <nil>}
2023-08-29T14:35:28.754-0600    DEBUG   server/server.go:195    Iterating commits in repository: https://github.com/jpmcb/dotfiles
2023-08-29T14:35:28.754-0600    DEBUG   server/server.go:207    Inspecting commit: jpmmcb@amazon.com 1f7099ac02c52ff4b775ac894b2e26ed5a56207e 2022-12-08 22:47:01 +0000 UTC
2023-08-29T14:35:28.757-0600    DEBUG   server/server.go:221    Checking if commit already in database: 1f7099ac02c52ff4b775ac894b2e26ed5a56207e
2023-08-29T14:35:28.760-0600    DEBUG   server/server.go:238    Finished processing: https://github.com/jpmcb/dotfiles

In the case that if fails, this should now return the invalid request back down to the client:

❯ pizza bake -e http://localhost:8080 github.com/jpmcb/dotfiles
Resp body: Could not normalize provided repo URL: repo URL missing valid protocol scheme (https, git, file): github.com/jpmcb/dotfiles
❯ pizza bake -e http://localhost:8080 https://github.com/jpmcb/doesntexist
Resp body: Error validating remote git repo URL: could not list remote repository: authentication required

(I thought the auth error was maybe a bug in go-git, but interestingly, this is the same behavior as git ls-remote and is working as expected).

What type of PR is this? (check all applicable)

Related Tickets & Documents

Closes #29 cc @k1nho

Mobile & Desktop Screenshots/Recordings

N/a

Added tests?

Added to documentation?

[optional] Are there any post-deployment tasks we need to perform?

[optional] What gif best describes this PR or how it makes you feel?

jpmcb commented 1 year ago

Force pushed to address linting issue.

k1nho commented 1 year ago

🤯 Amazing 👏👏👏👏