OpenTAXII cuts the XML messages which are longer than 64Kb

[zentavr]

When we import the messages from 3rd-paty sources, oftenly we have a situation when the XML message gets cut inside MySQL database.

Seems like the reason is because the MySQL cell is BLOB which is only 64Kb. The is not the mistake of the vendors of OpenTAXII, but the design of SQLAlchemy they use under the hood. SQLAlchemy is not going to change the internals of the library, but there is a workaround to make the life easier for MySQL users. We can use with_variants() method in order to tune the field for MySQL database.

See here.

[arcsector]

Putting this here for posterity:

For those wanting to migrate existing opentaxii databases to be mediumblob, here's the way i did it (mine took around 8 hours to finish with 150Gb of data, and i had to increase the buffer size in /etc/my.cnf to get it to work):

CREATE TABLE new_inbox_messages LIKE inbox_messages;
CREATE TABLE new_content_blocks LIKE content_blocks;

ALTER TABLE new_inbox_messages MODIFY COLUMN original_message mediumblob;
ALTER TABLE new_content_blocks MODIFY COLUMN content mediumblob;

INSERT INTO new_inbox_messages SELECT * FROM inbox_messages;
INSERT INTO new_content_blocks SELECT * FROM content_blocks;

RENAME TABLE inbox_messages TO old_inbox_messages;
RENAME TABLE new_inbox_messages TO inbox_messages;
RENAME TABLE content_blocks TO old_content_blocks;
RENAME TABLE new_content_blocks TO content_blocks;

ALTER TABLE content_blocks DROP FOREIGN KEY content_blocks_ibfk_1;
ALTER TABLE collection_to_content_block DROP FOREIGN KEY collection_to_content_block_ibfk_2;
ALTER TABLE `content_blocks` ADD CONSTRAINT `content_blocks_ibfk_1` FOREIGN KEY (`inbox_message_id`) REFERENCES `inbox_messages` (`id`) ON DELETE CASCADE ON UPDATE CASCADE;
ALTER TABLE `collection_to_content_block` ADD CONSTRAINT `collection_to_content_block_ibfk_2` FOREIGN KEY (`content_block_id`) REFERENCES `content_blocks` (`id`) ON DELETE CASCADE;

[traut]

the issue was fixed by provided patch