aptly-dev / aptly

aptly - Debian repository management tool
https://www.aptly.info/
MIT License
2.55k stars 369 forks source link

Fix publishing deadlock #1174

Closed randombenj closed 1 year ago

randombenj commented 1 year ago

Requirements

All new code should be covered with tests, documentation should be updated. CI should pass.

Description of the Change

A race condition for publishing packages and mirrors at the same time was introduced in commit 77d7c38.

The problem is that when opening a leveldb transaction and performing another 'put' to the db the system freezes.

See this sample code for explanation:

package main

import (
    "fmt"

    "github.com/syndtr/goleveldb/leveldb"
)

func main() {
    db, err := leveldb.OpenFile("whatever", nil)
    if err != nil {
        panic(err)
    }
    defer db.Close()

    transaction, err := db.OpenTransaction()
    if err != nil {
        panic(err)
    }
    defer transaction.Discard()

    fmt.Println("Putting A1")
    transaction.Put([]byte("A1"), []byte("value"), nil)

    fmt.Println("Putting A2 (non transaction)")
    db.Put([]byte("A2"), []byte("value"), nil) // <-- will freeze here!
    fmt.Println("Finished putting A2")

    fmt.Println("Committing transaction")
    err = transaction.Commit()
    if err != nil {
        panic(err)
    }
}

See also: https://github.com/syndtr/goleveldb/issues/431

The problem is actually not in goleveldb, but rather in the aptly pool lock: https://github.com/aptly-dev/aptly/blob/master/files/package_pool.go#L234

1) [THREAD-1] Creates transaction 2) [THREAD-2] Mirror publishes (pool gets locked) -> publish is waiting on db.Put(...) for transaction to finish. 3) [THREAD-1] Tries to lock the pool (wich is already locked by [THREAD-2] that is waiting for the current transaciton to finish). dead lock

Checklist

codecov[bot] commented 1 year ago

Codecov Report

Merging #1174 (8c8c70e) into master (1501a4e) will decrease coverage by 0.02%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1174      +/-   ##
==========================================
- Coverage   65.99%   65.97%   -0.02%     
==========================================
  Files         143      143              
  Lines       16196    16190       -6     
==========================================
- Hits        10688    10682       -6     
- Misses       4754     4756       +2     
+ Partials      754      752       -2     
Impacted Files Coverage Δ
deb/import.go 76.50% <100.00%> (+0.92%) :arrow_up:

... and 1 file with indirect coverage changes

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more