Program gets killed during compilation time

[kkc2894]

I am using the machine based on "malicious-shamir-party reading" and trying to read a file with approx 100M numbers. After reading each element I am storing in an array of sint type. Further for a query, I am searching the count of elements equal to search query using a==b command. This setup work up to 1k element but after that program gets killed.

[1]+ Killed Scripts/compile-run.py -E mal-shamir python_test -- -N 3

I am using docker images to run the system. Could you please help?

The underlying code:

def read_numbers_from_file(filename): with open(filename, 'r') as file: num_rows = 1000 numbers = Array(num_rows, sint) count = 0 while count < num_rows: line = file.readline() if line: numbers[count] = sint(int(line.strip())) count += 1 # Increment the counter else: break return numbers

filename = 'data_1M.csv' numbers = read_numbers_from_file(filename)

search_query=sint(1) count = sint(0) length=len(numbers)

def equality_test(): for i in range(len(numbers)): count.iadd(numbers[i] == search_query)

equality_test() print_ln('Count for %s is %s',search_query.reveal(), count.reveal())

I even tried the below version of the code:

program.use_edabit(True) def read_numbers(): numbers=sint.get_input_from(0, size=1000000) b = sint.get_input_from(1) eq_bits = [x == b for x in numbers] b_in_a = sum(eq_bits) print_ln("Count of %s is %s",b.reveal(), b_in_a.reveal())

read_numbers()

Party 0 has 1M numbers in Input_P0_0 file under Player_Data folder Party 1 has 1 number in Input_P1_0 file under Player_Data folder

[mkskeller]

This is because of the unrolling of large loops. In the second example, you should be able to fix this using

eq_bits = numbers == b
b_in_a = eq_bits.sum()

This switches to vectorized computation.

The first example is more difficult because it involves hard-coding all inputs in the program. I therefore recommend you stick to the second approach.

[kkc2894]

Thank you for quick response. When making the code changes as asked, I am getting below error: /usr/src/MP-SPDZ/Scripts/run-common.sh: line 90: 2363 Killed $my_prefix $SPDZROOT/$bin $i $params 2>&1 2364 Done | { if test "$BENCH"; then if test $i = $front_player; then tee -a $log; else cat >> $log; fi; else if test $i = $front_player; then tee $log; else cat > $log; fi; fi; }

My code: import time program.use_edabit(True) def read_numbers_from_file(): numbers = sint.get_input_from(0, size=1000000) b = sint.get_input_from(1) eq_bits = numbers == b b_in_a = eq_bits.sum() print_ln("Count of %s is %s", b.reveal(), b_in_a.reveal())

start_time = time.time() read_numbers_from_file() end_time = time.time()

[mkskeller]

What is the output of Scripts/memory-usage.py <program-with-args>? It might be that the memory requirements exceed your RAM.

[kkc2894]

I get time as printed as "Execution time:0.22 ms". However, the verbose of the run provides online/offline time. Which one should one consider to understand the time taken to run the code under MPSPDZ setting?

You cannot use the timing in the Python code because it doesn't measure the actual execution.

Also, MP-SPDZ considers only trusted client settings? Or Unstrusted client setting is also there?

I understand since I am using malicious majority honest setting based on Shamir's secret-sharing. if the number of parties is 3 parties, at least two have to be honest. But what about the clients who are communicating with these parties to do some computation? Is there any setting available for trusted or untrusted clients?

The standard client interface assumes an untrusted client, which means that it cannot influence the computation beyond the input provided and it does not learn more than the designated output.

[kkc2894]

A follow up question, so the online time is the time to consider the end-to-end latency? "Spent 0.96727 seconds (84.4956 MB, 22 rounds) on the online phase and 8.87888 seconds (396.995 MB, 2460 rounds) on the preprocessing/offline phase." --> Meaning 0.96727 seconds is the end-to-end time.

If not, how to get the total execution time per party?

[mkskeller]

It depends on what you mean by end-to-end. The line you mention splits up the time in several phases whereas "Time = ..." indicates the wall clock time for the computation, excluding some setup costs.