Implement larger message handling feature

[dewmal]

Description

There is currently a limitation in gossip pubsub that prevents sending data over 1MB. This restriction can potentially impact the functionality and performance of our system, especially when dealing with larger datasets or complex messages.

Current Behavior

• Gossip pubsub has a hard limit of 1MB for message size.
• Attempts to send data exceeding 1MB likely result in errors or message truncation.

Desired Behavior

We need to implement a solution to handle data transfers exceeding the 1MB limit. This could involve:

1. Message fragmentation and reassembly
2. Compression techniques
3. Alternative data transfer methods for large payloads

Proposed Solutions

1. Message Fragmentation: Split large messages into smaller chunks, each under 1MB, and reassemble them on the receiving end.
2. Data Compression: Implement compression algorithms to reduce the size of the data before transmission.
3. Hybrid Approach: Use gossip pubsub for metadata and small payloads, while employing a separate mechanism (e.g., direct P2P transfer) for larger data sets.

Tasks

• [x] #33
• [x] #34
• [x] #35
• [x] #36
• [x] #37

[dewmal]

Increase gossip pub sub message limit