search

hyperflow-wms / montage-workflow

Montage workflow port to HyperFlow
2 stars 3 forks source link

Statically Linked Executables #2

Open Zongshun96 opened 4 years ago

Zongshun96 commented 4 years ago

Hi, Do you have those statically linked version of those executables? I was trying to use those executables with your aws lambda executor. However, lambda functions can read those executables but would fail to run them. I guess it is because the linking issue discussed here. Thank you. Best, Zongshun Zhang

Zongshun96 commented 4 years ago

Hi, Do you have those statically linked executables?

Thanks, Zongshun Zhang

balis commented 4 years ago

Hi, can specifically describe what fails in the execution? As far as I remember Montage binaries are linked statically by default.

Zongshun96 commented 4 years ago

Oh, Sorry. I was trying the aws lambda executor in js. And lambda keeps saying error when run those executables, so I was wonderring if it was the linking issue.

Where can I ask about your JS Lambda executor? That repository seems not have an issue page?

I think I fixed 2 issues and stuck at running the executables. (I wonder if those 2 "fix" causes problems) First, nodejs 8 is not supported by lambda anymore so I changed this line to 

runtime: nodejs12.x

Second, it looks like there is a bug at here that if system env path was set, the prefix became the path variable of the system. I hard coded it to my prefix.

======================================== Then I made 2 attempts. If I put those executables beside the handler.js, the handler would try to find them in address "/var/task". While the runables should be at "/".

  | 2020-07-10T11:37:55.851-04:00 | 2020-07-10T15:37:55.850Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO Running /var/task/mProjectPP

If I put those executables in S3, handler simple log error to run them.

  | 2020-07-10T11:53:21.653-04:00 | 2020-07-10T15:53:21.653Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Running executable from S3
  | 2020-07-10T11:53:21.653-04:00 | 2020-07-10T15:53:21.653Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Running /tmp/mProjectPP
  | 2020-07-10T11:53:21.658-04:00 | 2020-07-10T15:53:21.657Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f ERROR Error!!mProjectPP{"errno":"ENOENT","code":"ENOENT","syscall":"spawn /tmp/mProjectPP","path":"/tmp/mProjectPP","spawnargs":["-X","-x","0.90307","2mass-atlas-001124n-j0890244.fits","p2mass-atlas-001124n-j0890244.fits","big_region_20180402_165339_22325.hdr"]}
  | 2020-07-10T11:53:21.658-04:00 | 2020-07-10T15:53:21.658Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO My exe close mProjectPP

======================================== The full log of one executable in first attempt from cloudwatch


  | 2020-07-10T11:37:55.649-04:00 | START RequestId: b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 Version: $LATEST
  | 2020-07-10T11:37:55.655-04:00 | 2020-07-10T15:37:55.655Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO Executable: mProjectPP
  | 2020-07-10T11:37:55.655-04:00 | 2020-07-10T15:37:55.655Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO Arguments: -X,-x,0.90415,2mass-atlas-001124n-j0870232.fits,p2mass-atlas-001124n-j0870232.fits,big_region_20180402_165339_22325.hdr
  | 2020-07-10T11:37:55.655-04:00 | 2020-07-10T15:37:55.655Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO Inputs: big_region_20180402_165339_22325.hdr,2mass-atlas-001124n-j0870232.fits
  | 2020-07-10T11:37:55.655-04:00 | 2020-07-10T15:37:55.655Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO Outputs: p2mass-atlas-001124n-j0870232.fits,p2mass-atlas-001124n-j0870232_area.fits
  | 2020-07-10T11:37:55.655-04:00 | 2020-07-10T15:37:55.655Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO Bucket: hyperflow
  | 2020-07-10T11:37:55.655-04:00 | 2020-07-10T15:37:55.655Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO Prefix: 0.25
  | 2020-07-10T11:37:55.656-04:00 | 2020-07-10T15:37:55.656Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO Downloading hyperflow/0.25/big_region_20180402_165339_22325.hdr
  | 2020-07-10T11:37:55.681-04:00 | 2020-07-10T15:37:55.681Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO Downloading hyperflow/0.25/2mass-atlas-001124n-j0870232.fits
  | 2020-07-10T11:37:55.739-04:00 | 2020-07-10T15:37:55.739Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO Downloaded file {"Bucket":"hyperflow","Key":"0.25/big_region_20180402_165339_22325.hdr"}
  | 2020-07-10T11:37:55.850-04:00 | 2020-07-10T15:37:55.850Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO Downloaded file {"Bucket":"hyperflow","Key":"0.25/2mass-atlas-001124n-j0870232.fits"}
  | 2020-07-10T11:37:55.850-04:00 | 2020-07-10T15:37:55.850Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO All files have been downloaded successfully
  | 2020-07-10T11:37:55.851-04:00 | 2020-07-10T15:37:55.850Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO Running /var/task/mProjectPP
  | 2020-07-10T11:37:55.856-04:00 | 2020-07-10T15:37:55.856Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 ERROR Error!!mProjectPP{"errno":"ENOENT","code":"ENOENT","syscall":"spawn /var/task/mProjectPP","path":"/var/task/mProjectPP","spawnargs":["-X","-x","0.90415","2mass-atlas-001124n-j0870232.fits","p2mass-atlas-001124n-j0870232.fits","big_region_20180402_165339_22325.hdr"]}
  | 2020-07-10T11:37:55.857-04:00 | 2020-07-10T15:37:55.857Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO My exe close mProjectPP
  | 2020-07-10T11:37:55.857-04:00 | 2020-07-10T15:37:55.857Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO Uploading hyperflow/0.25/p2mass-atlas-001124n-j0870232.fits
  | 2020-07-10T11:37:55.858-04:00 | 2020-07-10T15:37:55.858Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO Uploading hyperflow/0.25/p2mass-atlas-001124n-j0870232_area.fits
  | 2020-07-10T11:37:55.858-04:00 | 2020-07-10T15:37:55.858Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO Error reading file p2mass-atlas-001124n-j0870232.fits
  | 2020-07-10T11:37:55.860-04:00 | 2020-07-10T15:37:55.860Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO [Error: ENOENT: no such file or directory, open '/tmp/p2mass-atlas-001124n-j0870232.fits'] { errno: -2, code: 'ENOENT', syscall: 'open', path: '/tmp/p2mass-atlas-001124n-j0870232.fits' }
  | 2020-07-10T11:37:55.860-04:00 | 2020-07-10T15:37:55.860Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 ERROR A file failed to process
  | 2020-07-10T11:37:55.861-04:00 | 2020-07-10T15:37:55.860Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 ERROR Error: Error uploading
  | 2020-07-10T11:37:55.861-04:00 | 2020-07-10T15:37:55.861Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO Error reading file p2mass-atlas-001124n-j0870232_area.fits
  | 2020-07-10T11:37:55.862-04:00 | 2020-07-10T15:37:55.862Z b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 INFO [Error: ENOENT: no such file or directory, open '/tmp/p2mass-atlas-001124n-j0870232_area.fits'] { errno: -2, code: 'ENOENT', syscall: 'open', path: '/tmp/p2mass-atlas-001124n-j0870232_area.fits' }
  | 2020-07-10T11:37:55.862-04:00 | END RequestId: b123aae4-e295-4aca-9a0b-99f3f5dd2bd6
  | 2020-07-10T11:37:55.862-04:00 | REPORT RequestId: b123aae4-e295-4aca-9a0b-99f3f5dd2bd6 Duration: 209.54 ms Billed Duration: 300 ms Memory Size: 2048 MB Max Memory Used: 35 MB

The full log of second attempt

  | 2020-07-10T11:53:21.424-04:00 | START RequestId: 7ad12a80-672d-4ca7-bfaf-703b87778c9f Version: $LATEST
  | 2020-07-10T11:53:21.429-04:00 | 2020-07-10T15:53:21.428Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Executable: mProjectPP
  | 2020-07-10T11:53:21.429-04:00 | 2020-07-10T15:53:21.428Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Arguments: -X,-x,0.90307,2mass-atlas-001124n-j0890244.fits,p2mass-atlas-001124n-j0890244.fits,big_region_20180402_165339_22325.hdr
  | 2020-07-10T11:53:21.429-04:00 | 2020-07-10T15:53:21.428Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Inputs: big_region_20180402_165339_22325.hdr,2mass-atlas-001124n-j0890244.fits
  | 2020-07-10T11:53:21.429-04:00 | 2020-07-10T15:53:21.428Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Outputs: p2mass-atlas-001124n-j0890244.fits,p2mass-atlas-001124n-j0890244_area.fits
  | 2020-07-10T11:53:21.429-04:00 | 2020-07-10T15:53:21.428Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Bucket: hyperflow
  | 2020-07-10T11:53:21.429-04:00 | 2020-07-10T15:53:21.428Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Prefix: 0.25
  | 2020-07-10T11:53:21.436-04:00 | 2020-07-10T15:53:21.429Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Downloading hyperflow/0.25/big_region_20180402_165339_22325.hdr
  | 2020-07-10T11:53:21.455-04:00 | 2020-07-10T15:53:21.455Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Downloading hyperflow/0.25/2mass-atlas-001124n-j0890244.fits
  | 2020-07-10T11:53:21.457-04:00 | 2020-07-10T15:53:21.457Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Downloading hyperflow/0.25/mProjectPP
  | 2020-07-10T11:53:21.510-04:00 | 2020-07-10T15:53:21.509Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Downloaded file {"Bucket":"hyperflow","Key":"0.25/big_region_20180402_165339_22325.hdr"}
  | 2020-07-10T11:53:21.568-04:00 | 2020-07-10T15:53:21.568Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Downloaded file {"Bucket":"hyperflow","Key":"0.25/2mass-atlas-001124n-j0890244.fits"}
  | 2020-07-10T11:53:21.653-04:00 | 2020-07-10T15:53:21.652Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Downloaded executable {"Bucket":"hyperflow","Key":"0.25/mProjectPP"}
  | 2020-07-10T11:53:21.653-04:00 | 2020-07-10T15:53:21.653Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO All files have been downloaded successfully
  | 2020-07-10T11:53:21.653-04:00 | 2020-07-10T15:53:21.653Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Running executable from S3
  | 2020-07-10T11:53:21.653-04:00 | 2020-07-10T15:53:21.653Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Running /tmp/mProjectPP
  | 2020-07-10T11:53:21.658-04:00 | 2020-07-10T15:53:21.657Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f ERROR Error!!mProjectPP{"errno":"ENOENT","code":"ENOENT","syscall":"spawn /tmp/mProjectPP","path":"/tmp/mProjectPP","spawnargs":["-X","-x","0.90307","2mass-atlas-001124n-j0890244.fits","p2mass-atlas-001124n-j0890244.fits","big_region_20180402_165339_22325.hdr"]}
  | 2020-07-10T11:53:21.658-04:00 | 2020-07-10T15:53:21.658Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO My exe close mProjectPP
  | 2020-07-10T11:53:21.658-04:00 | 2020-07-10T15:53:21.658Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Uploading hyperflow/0.25/p2mass-atlas-001124n-j0890244.fits
  | 2020-07-10T11:53:21.659-04:00 | 2020-07-10T15:53:21.659Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Uploading hyperflow/0.25/p2mass-atlas-001124n-j0890244_area.fits
  | 2020-07-10T11:53:21.659-04:00 | 2020-07-10T15:53:21.659Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Error reading file p2mass-atlas-001124n-j0890244.fits
  | 2020-07-10T11:53:21.672-04:00 | 2020-07-10T15:53:21.672Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO [Error: ENOENT: no such file or directory, open '/tmp/p2mass-atlas-001124n-j0890244.fits'] { errno: -2, code: 'ENOENT', syscall: 'open', path: '/tmp/p2mass-atlas-001124n-j0890244.fits' }
  | 2020-07-10T11:53:21.672-04:00 | 2020-07-10T15:53:21.672Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f ERROR A file failed to process
  | 2020-07-10T11:53:21.672-04:00 | 2020-07-10T15:53:21.672Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f ERROR Error: Error uploading
  | 2020-07-10T11:53:21.673-04:00 | 2020-07-10T15:53:21.673Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO Error reading file p2mass-atlas-001124n-j0890244_area.fits
  | 2020-07-10T11:53:21.673-04:00 | 2020-07-10T15:53:21.673Z 7ad12a80-672d-4ca7-bfaf-703b87778c9f INFO [Error: ENOENT: no such file or directory, open '/tmp/p2mass-atlas-001124n-j0890244_area.fits'] { errno: -2, code: 'ENOENT', syscall: 'open', path: '/tmp/p2mass-atlas-001124n-j0890244_area.fits' }
  | 2020-07-10T11:53:21.673-04:00 | END RequestId: 7ad12a80-672d-4ca7-bfaf-703b87778c9f
  | 2020-07-10T11:53:21.673-04:00 | REPORT RequestId: 7ad12a80-672d-4ca7-bfaf-703b87778c9f Duration: 247.22 ms Billed Duration: 300 ms Memory Size: 2048 MB Max Memory Used: 48 MB

Thanks,

Zongshun96 commented 4 years ago

Hi, Were you able to reproduce the error? If it is not the right place to ask, can you tell me who I should contact about this error?

Thanks,