Testing examples in plaingraph_test.cpp, most of them cannnot work

[ghost]

I generate kron_10_10 use your g500_generator and test examples in plaingraph_test.cpp, only -j 0 and -j 1 run well.

Here are the results of others:

for -j 2 and -j 3

./graphone64 -i kron_10_10/ -o ./b/ -j 3 -v 10240 
input dir = kron_10_10/
output dir = ./b/
Global vcount = 10240
Threads Count = 7
huge page alloc failed while reading input dir
Reading 1 file time = 0.000270128
End marker = 4969
 total edge count8192
edge counts in basefile = 1
Segmentation fault (core dumped) 

and using graph32 will cause terminal stuck like

./graphone32 -i kron_10_10/ -o ./b/ -j 3 -v 10240 
input dir = kron_10_10/
output dir = ./b/
Global vcount = 10240
Threads Count = 7
huge page alloc failed while reading input dir
Reading 1 file time = 0.000507116
End marker = 9939
 total edge count16384
edge counts in basefile = 1
^C

for -j 4

./graphone64 -i kron_10_10/ -o ./b/ -j 4 -v 10240 
input dir = kron_10_10/
output dir = ./b/
Global vcount = 10240
Threads Count = 7
batch_size = 1
batch_size = 1
edge_count = 32
new_count  = 32
graphone64: /mnt/d/work/c/GraphOne/test/plain_to_edge.h:236: void* recovery_func(void*) [with T = dst_id_t]: Assertion `0' failed.
Aborted (core dumped)

 ./graphone32 -i kron_10_10/ -o ./b/ -j 4 -v 10240 
input dir = kron_10_10/
output dir = ./b/
Global vcount = 10240
Threads Count = 7
batch_size = 1
batch_size = 1
edge_count = 64
new_count  = 64
graphone32: /mnt/d/work/c/GraphOne/test/plain_to_edge.h:236: void* recovery_func(void*) [with T = dst_id_t]: Assertion `0' failed.
Aborted (core dumped)

-j {50...56} has similar problem

I want to redo your work in the paper of FAST'19. Would you like to instruct me how to fix this? Or could you share me a useable old version?

[pradeep-k]

Many of the tests in plaingraph_test.cpp are internal tests as well.
And some of them require usage of -r flag. I am updating the documentation to reflect this. Then, you will be able to run those cases.

[pradeep-k]

91c3a7c8c0505f34fd8c717d2072f4258f845e3f fixes bug, and 7cff7a3d8333702d96f5d67757a1db662f966df0 updates the documents.

Kindly, let me know if you are able to run various testcases.

[ghost]

Thanks you for your updates. For -j 2, I found that small graph kron-10 cause segfault but kron-21 works well. For -j 4, both work well. -j 3 still cause segfault. Since it names mix, I think it can be achieved by other testcases.

There is another question. When i run the code, i get the huge page alloc failed ... warnings. Does it affect the running speed or result of the program?

[pradeep-k]

There is another question. When i run the code, i get the huge page alloc failed ... warnings. Does it affect the running speed or result of the program?

No, it does not impact. We try to allocate huge pages at only few places. Other places I have commented because a few users with different Linux flavors were facing compilation issues.

-j3 is working for me. Just make sure what kind of graph you have. I have put some assert for text files. I am only supporting it for binary files. What is your graph type, text or binary file? Also, pay attention to -r value.

I will check -j2 value for small graphs. What value of -r did you choose.

Can you send me your command line for -j2 and -j4 cases?

[ghost]

There is another question. When i run the code, i get the huge page alloc failed ... warnings. Does it affect the running speed or result of the program?

No, it does not impact. We try to allocate huge pages at only few places. Other places I have commented because a few users with different Linux flavors were facing compilation issues.

-j3 is working for me. Just make sure what kind of graph you have. I have put some assert for text files. I am only supporting it for binary files. What is your graph type, text or binary file? Also, pay attention to -r value.

I will check -j2 value for small graphs. What value of -r did you choose.

Can you send me your command line for -j2 and -j4 cases?

I have noticed the prompt of using binary, so i used binary file. ./graphone32 -i kron_10_10/bin/ -j 2 -v 2097152 -s 1 -r 10 -r 5..10 causes segfault. Inside the bin folder, I only put one file named kron-21-16, generated by your tool.

-j 4 works well. The .log file used is the output of -j 1 , same as your example in README: ./graphone32 -j 4 -v 2097152 -s 1 -i ./out/friend.elog -r 16

Is -j 3 the one that uses both edge log and adjacency list? I think I guess the wrong meaning of its name.

[pradeep-k]

Thanks for your feedback and command. The bug has been fixed by commit number: 5531a3a701d773064a3b5f974f00809480f0d61d

Kindly check again and let me know.

[pradeep-k]

For -j3 meaning: Assume you have 100 edges, but only 90 edges has been converted to adjacency list. But you are interested in running analytics on all the 100 edges. One option is to let user use both adjacency list (of 90 edges) plus edge list (of 10 edges).
This is called data visibility in the paper. There are other ways to achieve simple data visibility as well.

[ghost]

Thanks for your feedback and command. The bug has been fixed by commit number: 5531a3a

Kindly check again and let me know.

Nice work, problem fixed.

[ghost]

For -j3 meaning: Assume you have 100 edges, but only 90 edges has been converted to adjacency list. But you are interested in running analytics on all the 100 edges. One option is to let user use both adjacency list (of 90 edges) plus edge list (of 10 edges). This is called data visibility in the paper. There are other ways to achieve simple data visibility as well.

I'm still little confused of the testcases. what's the difference between test_ingestion and test_logging? I think both of them read the input files and convert them into edge log. One difference is test_logging can generate durable edge log file. I think test_archive is the procedure that converts the edge log into adjacency list. As you statement about test_mix, it should take the input of both edge list and adjacency list, but it only takes the raw input file. Besides, at the end of these testcases, there is run_bfs() to perform graph analysis. What's the differences among the data storage situations of these three testcases?

[pradeep-k]

All of them varies on how many of edges have been converted to adjacency list, and how many of them are not.

test_ingestion: converts all the edges present in the edge log to adjacency list. Table 3, column ingestion results in the FAST paper. test_logging: Keeps all the edges in the edge log itself. Does not convert any edges to adjacency list. Table, column logging results in the FAST paper. -j2 : it is the archiving rate at a specific arhiving threshold. Table 3, column archiving, and archiving rate in Fig. 13. -j3: When we only convert some edges of the edge log to adjacency list, and leave rest of them in the edge log. Fig. 13 except archiving rate shows this impact on the analytics performance.

Archiving is the process to convert edge log to adjacency list, logging is defined as logging the incoming stream data (in this case from text/binary files) to an in-memory edge log. Durability is achieved by write the edge log data to disk periodically. Hence, the input data is always same in all of the testcases.

Because of data-visibility concept, the edge log data will only periodically be converted to adjacency list format, but an analytic can choose to work at any update level (granularity), thus it can choose to work on whatever adjacency list is ready at this moment, plus non-archived edges. If an analytic is only interested in adjacency list format, it is high possibility that it might be slightly stale. We support this using STALE flag in graph view. There are many other flags in the graph view creation that makes the graph view API very powerful yet very simple.

[ghost]

Thank you for your detail explaination!