Treat everything as IPv6 address: https://en.wikipedia.org/wiki/IPv6#Software (larger, but fixed-width payloads). IPv4 addresses can be left-padded with 80-bits of 0s and 16-bits of 1s.
Store as a NumPy structured array of two uint64s. We need 128 bits to store an IPv6 address, so we'll store the lower 64 bits in a uint64 and the upper 128bits in a uint64