robur-coop / albatross

Albatross: orchestrate and manage MirageOS unikernels with Solo5
ISC License
143 stars 17 forks source link

Out of memory in albatross-console #199

Closed reynir closed 18 minutes ago

reynir commented 6 hours ago

Possibly related to #198 and #194

Nov 20 09:09:13 bark.sorbus.is albatross-console[953]: albatross-console: [ERROR] exception Unix.Unix_error(Unix.EBADF, "check_descriptor", "") while writing
Nov 20 09:09:13 bark.sorbus.is albatross-console[953]: albatross-console: [ERROR] exception Unix.Unix_error(Unix.EBADF, "check_descriptor", "") while writing
Nov 20 09:09:13 bark.sorbus.is albatross-console[953]: albatross-console: [ERROR] exception Unix.Unix_error(Unix.EBADF, "check_descriptor", "") while writing
Nov 20 09:09:13 bark.sorbus.is albatross-console[953]: albatross-console: [ERROR] exception Unix.Unix_error(Unix.EBADF, "check_descriptor", "") while writing
Nov 20 09:09:13 bark.sorbus.is albatross-console[953]: albatross-console: [ERROR] :opam-mirror error while reading Out of memory

I will try run the unikernel by hand and see if the output is particularly spammy. I only seem to get one line from albatross-console:

console 2024-11-20T09:05:44-00:00:             |      ___|

I think this is the first line of the solo5 ASCII logo:

                |      ___|
      __|  _ \  |  _ \ __ \
    \__ \ (   | | (   |  ) |
    ____/\___/ _|\___/____/
hannesm commented 4 hours ago

this looks like something is wrong in the data structures... how many console readers do you have/had? 10s or 100s or 1000s?

reynir commented 3 hours ago

I had recently restarted albatross, so I think it was less than 10 readers that had connected.

reynir commented 2 hours ago

I made the Lwt.catch handler reraise the exception and got this:

Nov 20 12:15:39 bark.sorbus.is albatross-console[105714]: Fatal error: exception Out of memory
Nov 20 12:15:39 bark.sorbus.is albatross-console[105714]: Raised by primitive operation at Dune__exe__Albatross_console.read_console.(fun).loop.(fun).f.(fun).update.(fun) in file "daemon/albatross_console.ml", line 41, characters 58-67
Nov 20 12:15:39 bark.sorbus.is albatross-console[105714]: Called from Stdlib__List.find_all.find in file "list.ml", line 242, characters 17-20
Nov 20 12:15:39 bark.sorbus.is albatross-console[105714]: Called from Dune__exe__Albatross_console.read_console.(fun).loop.(fun).f.(fun).update in file "daemon/albatross_console.ml", line 41, characters 24-70
Nov 20 12:15:39 bark.sorbus.is albatross-console[105714]: Called from Stdlib__Map.Make.update in file "map.ml", line 283, characters 18-28
Nov 20 12:15:39 bark.sorbus.is albatross-console[105714]: Called from Dune__exe__Albatross_console.read_console.(fun).loop.(fun).f.(fun) in file "daemon/albatross_console.ml", line 47, characters 23-69
Nov 20 12:15:39 bark.sorbus.is albatross-console[105714]: Called from Lwt.Sequential_composition.map.(fun) in file "src/core/lwt.ml", line 1997, characters 28-33
Nov 20 12:15:39 bark.sorbus.is albatross-console[105714]: Re-raised at Dune__exe__Albatross_console.read_console.(fun) in file "daemon/albatross_console.ml", line 61, characters 18-33
Nov 20 12:15:39 bark.sorbus.is albatross-console[105714]: Called from Lwt.Sequential_composition.catch.create_result_promise_and_callback_if_deferred.callback in file "src/core/lwt.ml", line 2041, characters 16-21

The primitive operation is fd <> fd'. So I guess it's not safe to compare file descriptors using the polymorphic comparison after all /o\

hannesm commented 20 minutes ago

is this as well fixed by #200?

reynir commented 19 minutes ago

Yes!

ons. 20. nov. 2024 15.56 skrev Hannes Mehnert @.***>:

is this as well fixed by #200 https://github.com/robur-coop/albatross/pull/200?

— Reply to this email directly, view it on GitHub https://github.com/robur-coop/albatross/issues/199#issuecomment-2488803773, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACVC33NC5IHJ5CHUOK2WKT2BSPKLAVCNFSM6AAAAABSECOIH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBYHAYDGNZXGM . You are receiving this because you were assigned.Message ID: @.***>