the work on block partitioning was done last summer & fall, and had to be set aside to work on other more pressing projects. it was almost working when it was set aside. Took the time so that it works now:
TEST RESULTS
test 1 success: Log perms confirmed
| content of subdirs of /home/peter/sarra_devdocroot |
test 2 success: compare contents of downloaded_by_sub_amqp and downloaded_by_sub_cp are the same
test 3 success: compare contents of downloaded_by_sub_cp and downloaded_by_sub_rabbitmqtt are the same
test 4 success: compare contents of downloaded_by_sub_rabbitmqtt and downloaded_by_sub_u are the same
test 5 success: compare contents of downloaded_by_sub_u and posted_by_shim are the same
test 6 success: compare contents of downloaded_by_sub_amqp and linked_by_shim are the same
test 7 success: compare contents of posted_by_shim and sent_by_tsource2send are the same
test 8 success: compare contents of downloaded_by_sub_amqp and cfile are the same
test 9 success: compare contents of cfile and cfr are the same
broker state:
source tree: total:1418 files: 1111 directories:307 rejected:10
| dd.weather routing |
test 10 success: post (9304) t_dd1 should have the same number of items as t_dd2 (9304)
test 11 success: sarra (18608) should receive the same number of items as both post (18608)
test 12 success: sarra (9304) should publish the same number of items as one post (9304)
test 13 success: sarra (9304) should winnow the same number of items as one post (9304)
test 14 success: subscribe (1418) should rx the same number of items as in static tree (1418)
| watch routing |
test 15 success: watch (1419) should be the same as subscribe amqp_f30 (1418)
test 16 success: sender (1419) should publish the same number of items as watch (1419)
test 17 success: rabbitmqtt (1419) should download same number of items as watch (1419)
test 18 success: subscribe u_sftp_f60 (1419) should download same number of items as sender (1419)
test 19 success: subscribe cp_f61 (1419) should download same number of items as sender (1419)
| poll routing |
test 20 success: poll sftp_f62 (1418) should publish same number of items of sender sent (1419)
test 21 success: poll sftp_f63 (1418) should see the same number of items as poll sftp_f62 posted (1419)
test 22 success: subscribe q_f71 (1418) should download same number of items as poll test1_f62 (1418)
| flow_post routing |
test 23 success: post test2_f61 (1111) should have the same number of files of sender (1111)
test 24 success: subscribe ftp_f70 (1111) should have the same number of items as post test2_f61 (1111)
test 25 success: post test2_f61 (1111) should post about the same number of files as shim_f63 (1111)
test 26 success: post test2_f61 (1111) should post about the same number of links as shim_f63 (1111)
test 27 success: static tree (307) directories should be posted twice: for 1st copy and linked_dir by shim_f63 (614)
| py infos routing |
test 28 success: 0 messages received that we don't know what happened.
| C routing |
test 29 success: cpost both pelles should post the same number of messages (1420) (1420)
test 30 success: cdnld_f21 subscribe downloaded (1111) the same number of files that was published by both van_14 and van_15 (1111)
test 31 success: veille_f34 should post as many files (1111) as subscribe cdnld_f21 downloaded (1111)
test 32 success: veille_f34 should post as many files (1111) as subscribe cfile_f44 downloaded (1111)
Overall partitioned_flow 32 of 32 passed (sample size: 1418) !
Most of the work was that the static_flow test, on which this is based, had evolved with many changes in the python.
What the test does...
sets a partition size of 200 bytes,
posts all the files from the static data tree in blocks of that size.
it is download in partitions by sarra/download_f20
which is then subscribed by amqp_f30
the subscribe has flowcb/block_reassembly active, and rebuilds the files.
watch etc... are looking at the products of the re-build and other operations done.
Getting this working also involved a number of fixes to the python that are in a corresponding branch/PR
the work on block partitioning was done last summer & fall, and had to be set aside to work on other more pressing projects. it was almost working when it was set aside. Took the time so that it works now:
Most of the work was that the static_flow test, on which this is based, had evolved with many changes in the python.
What the test does...
Getting this working also involved a number of fixes to the python that are in a corresponding branch/PR