Closed WarrDaddy closed 1 year ago
fyoorer
commented
1 year ago @WarrDaddy , can you try by providing absolute path to the -i and -o flags instead of short paths like ~/Desktop/compiled.txt.
Your command -c also looks wrong to me. you have to specify the placeholder {INPUT} somewhere for providing the chunk as input to the tool. There is no ~/Desktop/resolvers.txt inside the container too. take a look at the wiki for few examples
WarrDaddy
commented
1 year ago Hey there @fyoorer. Thanks for the response. I've corrected my command and the issue still exist.
user@Administrators-MacBook-Pro-7 ~/arsenal/ShadowClone [23:38:42]
(recon-venv) > $ python shadowclone.py -i /Users/user/Desktop/compiled.txt --split 3000000 -o /Users/user/Desktop/shadowoutput.txt -c "/go/bin/puredns resolve {INPUT} --resolvers /function/resolvers.txt" [±main ●●]
2023-02-11 23:39:11,778 [INFO] Splitting input file into chunks of 3000000 lines
2023-02-11 23:39:14,301 [INFO] Uploading chunks to storage
2023-02-11 23:39:14,469 [INFO] lithops.config -- Lithops v2.6.0
2023-02-11 23:39:14,480 [INFO] lithops.storage.backends.aws_s3.aws_s3 -- S3 client created - Region: us-west-1
2023-02-11 23:39:14,951 [INFO] lithops.serverless.backends.aws_lambda.aws_lambda -- AWS Lambda client created - Region: us-west-1
2023-02-11 23:39:14,954 [INFO] lithops.invokers -- ExecutorID 170467-0 | JobID M000 - Selected Runtime: lithops_v2-6-0_gupt/sc-runtime - 512MB
2023-02-11 23:39:16,651 [INFO] lithops.invokers -- ExecutorID 170467-0 | JobID M000 - Starting function invocation: execute_command() - Total: 4 activations
2023-02-11 23:39:16,655 [INFO] lithops.invokers -- ExecutorID 170467-0 | JobID M000 - View execution logs at /private/var/folders/q2/v0knyw_n45598tj6xm9clf4r0000gp/T/lithops/logs/170467-0-M000.log
2023-02-11 23:39:16,656 [INFO] lithops.wait -- ExecutorID 170467-0 - Getting results from 4 function activations
0%| | 0/4
2023-02-11 23:39:21,678 [INFO] lithops.executors -- ExecutorID 170467-0 - Cleaning temporary data
2023-02-11 23:39:21,683 [ERROR] Could not execute the runtime.Log poll output
(recon-venv) > $ lithops logs poll [±main ●●]
Activation: 'lithops_v2-6-0_gupt/sc-runtime' (fc364e0d-2132-4594-8da7-5281ad7f2a20)
[
2023-02-12 07:38:28,561 [INFO] lithops.worker.handler -- Lithops v2.6.0 - Starting AWS Lambda execution
2023-02-12 07:38:28,561 [INFO] lithops.worker.handler -- Execution ID: e90409-0-M000/00000
2023-02-12 07:38:28,584 [INFO] lithops.storage.backends.aws_s3.aws_s3 -- S3 client created - Region: us-west-1
2023-02-12 07:38:29,488 [INFO] lithops.worker.jobrunner -- Getting dataset from aws_s3://runtime-shadowclone/1a6d7cc8-ac31-44fb-b650-22699f5e927a
2023-02-12 07:38:29,612 [INFO] lithops.worker.jobrunner -- Going to execute 'execute_command()'
---------------------- FUNCTION LOG ----------------------
2023-02-12 07:38:41,270 [ERROR] lithops.worker.handler -- No completion message received from JobRunner process
----------------------- EXCEPTION !-----------------------
Traceback (most recent call last):
File "/function/lithops/worker/handler.py", line 202, in run_job
raise MemoryError('HANDLER', msg)
MemoryError: ('HANDLER', 'Function exceeded maximum memory and was killed')
----------------------------------------------------------
]
Activation: 'lithops_v2-6-0_gupt/sc-runtime' (4a47fb12-732a-4c28-80a5-0ad741ed78e1)
[
2023-02-12 07:39:17,002 [INFO] lithops.worker.handler -- Lithops v2.6.0 - Starting AWS Lambda execution
2023-02-12 07:39:17,002 [INFO] lithops.worker.handler -- Execution ID: 170467-0-M000/00003
2023-02-12 07:39:17,008 [INFO] lithops.storage.backends.aws_s3.aws_s3 -- S3 client created - Region: us-west-1
2023-02-12 07:39:17,849 [INFO] lithops.worker.jobrunner -- Getting dataset from aws_s3://runtime-shadowclone/652a70ec-2c67-4fe8-ab64-ea5c461cce67
2023-02-12 07:39:17,961 [INFO] lithops.worker.jobrunner -- Going to execute 'execute_command()'
---------------------- FUNCTION LOG ----------------------
----------------------- EXCEPTION !-----------------------
Traceback (most recent call last):
File "/function/lithops/worker/jobrunner.py", line 228, in run
result = func(**data)
File "shadowclone.py", line 94, in execute_command
AttributeError: 'NoneType' object has no attribute 'replace'
----------------------------------------------------------
2023-02-12 07:39:18,986 [INFO] lithops.worker.jobrunner -- Process finished
]
@WarrDaddy should be fixed now, can you give it another try. Please let me know if you still encounter these errors
Hello, there was an issue while running shadowclone.py to resolve dns records. From the terminal output below, you can see that 4 functions activated however it looks like the service was unable to execute the run time. Upon inspection of poll output, there was an exception within runnerjob.py.
Exception
Terminal Output
Poll output