service gets stuck and doesn't update anymore after some time of uninterrupted operation

paolobarbolini commented 3 years ago

After a few weeks of deps.rs running without new deploys or restarts it looks like the updater for popular crates and popular repos gets stuck and doesn't update anymore. This has happened multiple times now.

This is easily noticeable when serde releases a new version, since the homepage keeps pointing to the older version, causing the badge to report 1 outdated dependency (serde imports an exact version of serde-derive).

deps.rs

remote

local instance

local

Feliix42 commented 3 years ago

Any ideas on how this can happen? Maybe a caching issue?

robjtede commented 3 years ago

Anything useful logged? Does restarting the deployment help?

paolobarbolini commented 3 years ago

Does restarting the deployment help?

It does for a few weeks and then it happens again.

Feliix42 commented 3 years ago

Currently, everything is up to date (presumably due to the merge of #107). Let's see, how long it lasts.

paolobarbolini commented 3 years ago

It happened again

Feliix42 commented 2 years ago

So I guess it makes sense to assume that both #156 and #145 are somehow related to this?

As @kid pointed out, the logs show only a few error messages regarding timeouts, but nothing beyond that. Maybe it makes sense to try and locate the problem methodically:

[ ] How does the deployment machine look after deps has been running a while?
- [ ] Is there a slow but steady increase in memory usage for instance?
- [ ] Are we accidentally keeping any connections open?
[ ] Can we get a more detailed error logging set up? Or can we get a full log that spans the time before and after the error start occuring? Sadly the instance restart also means we lost the logs, I think.
[ ] Can we try to reproduce the problem locally on our machines by spamming it for a while with requests?
[ ] @cecton pointed out that we could be rate-limited. Can we check that? Since the log snippet posted by @kid shows both Gitlab and Github timeout-ing, I would guess it's not on the receiving end of our requests at least.

kid commented 2 years ago

definitely looks like a memory leak

cecton commented 2 years ago

If we were hitting the memory limit the process would have been killed with signal 9 and I would expect the thing to restart automatically and to be working again.

robjtede commented 2 years ago

Couldn't it just be the cache getting bigger over time, as we'd expect?

crjeder commented 2 years ago

For me it currently looks like this: And the badge for my crate does time out, too.

crjeder commented 2 years ago

For me it currently looks like this: And the badge for my crate does time out, too.

works again!

mumbleskates commented 8 months ago

there it goes again

paolobarbolini commented 5 months ago

I've tried running deps.rs locally with the following patches:

deps.rs

diff --git a/Cargo.lock b/Cargo.lock
index 9721623..4aeff6a 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -245,8 +245,6 @@ dependencies = [
 [[package]]
 name = "crates-index"
 version = "2.10.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8467f424ab6d70e5daf384289fa68247787effc13901dd0ca46bc9b4a62f1474"
 dependencies = [
  "gix",
  "hex",
diff --git a/Cargo.toml b/Cargo.toml
index 746daaf..d8f3589 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -16,7 +16,7 @@ badge = { path = "./libs/badge" }

 anyhow = "1"
 cadence = "1"
-crates-index = { version = "2", default-features = false, features = ["git"] }
+crates-index = { path = "../rust-crates-index", default-features = false, features = ["git"] }
 derive_more = "0.99"
 font-awesome-as-a-crate = "0.3"
 futures-util = { version = "0.3", default-features = false, features = ["std"] }

rust-crates-index

diff --git a/src/error.rs b/src/error.rs
index 7ae1d27..5f06ee2 100644
--- a/src/error.rs
+++ b/src/error.rs
@@ -7,7 +7,7 @@ pub use toml::de::Error as TomlDeError;
 #[derive(Debug, thiserror::Error)]
 #[allow(missing_docs)]
 pub enum Error {
-    #[error("\"gix\" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/`")]
+    #[error("\"gix\" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` {0:?}")]
     #[cfg(feature = "git")]
     Git(#[from] GixError),
     #[error("{0}")]

The logs were the following:

May 26 16:20:53.511 INFO Server running on port 8080
May 26 16:41:13.861 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` PrepareFetch(RefMap(Handshake(Transport(Io(Custom { kind: Other, error: "error sending request for url (https://github.com/rust-lang/crates.io-index/info/refs?service=git-upload-pack)" })))))
May 26 16:42:33.871 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` PrepareFetch(RefMap(Handshake(Transport(Io(Custom { kind: Other, error: "error sending request for url (https://github.com/rust-lang/crates.io-index/info/refs?service=git-upload-pack)" })))))
May 26 16:46:33.881 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` PrepareFetch(RefMap(Handshake(Transport(Io(Custom { kind: Other, error: "error sending request for url (https://github.com/rust-lang/crates.io-index/info/refs?service=git-upload-pack)" })))))
May 26 16:51:33.890 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` PrepareFetch(RefMap(Handshake(Transport(Io(Custom { kind: Other, error: "error sending request for url (https://github.com/rust-lang/crates.io-index/info/refs?service=git-upload-pack)" })))))
May 26 16:57:33.904 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` PrepareFetch(RefMap(Handshake(Transport(Io(Custom { kind: Other, error: "error sending request for url (https://github.com/rust-lang/crates.io-index/info/refs?service=git-upload-pack)" })))))
May 26 16:58:33.921 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` PrepareFetch(RefMap(Handshake(Transport(Io(Custom { kind: Other, error: "error sending request for url (https://github.com/rust-lang/crates.io-index/info/refs?service=git-upload-pack)" })))))
May 26 17:08:54.265 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:09:14.315 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:09:34.308 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:09:54.263 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:10:14.286 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:10:34.286 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:10:54.309 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:11:14.261 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:11:34.265 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:12:13.930 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` PrepareFetch(RefMap(Handshake(Transport(Io(Custom { kind: Other, error: "error sending request for url (https://github.com/rust-lang/crates.io-index/info/refs?service=git-upload-pack)" })))))
May 26 17:12:14.285 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:12:34.282 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:12:54.274 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:13:14.306 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:13:34.292 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:13:54.299 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:14:14.338 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:14:34.272 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))
May 26 17:14:54.340 ERRO failed refreshing the crates.io-index, the operation will be retried: "gix" crate failed. If problems persist, consider deleting `~/.cargo/registry/index/github.com-1ecc6299db9ec823/` Fetch(Negotiate(ObtainRefDuringIteration(NotFound { oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb), name: "refs/heads/master" })))

robjtede commented 5 months ago

ObtainRefDuringIteration(
  NotFound {
    oid: Sha1(4c381571c3c2bb1d61666f309be302f55ee84acb),
    name: "refs/heads/master"
  }
)

This reads to me like it's having issues with the repo collapses.

I ran current master locally and saw the same kind of errors as you, Paolo. Following the advice of the error message (deleting ~/.cargo/registry/index/github.com-...) I'm yet to see it resurface.

Possible solutions and workarounds:

we run rm -rf ~/.cargo/registry/index/github* as part of the restart cronjob
we refactor the service to use the sparse registry

robjtede commented 5 months ago

refactoring to sparse index wasn't too much work: https://github.com/deps-rs/deps.rs/pull/227

deps-rs / deps.rs