realm / aws-devicefarm

Github action for triggering runs on AWS devicefarm
MIT License
17 stars 10 forks source link

`test-application` does not set `output.arn` is not set if fails #16

Open gmazzo opened 1 year ago

gmazzo commented 1 year ago

I'm currently using ${{ steps.adf.outputs.arn }} with

      - id: adf
        name: Amazon Device Farm
        uses: realm/aws-devicefarm/test-application@v1
        with:
            ...

to reconstruct the test run URL to provide feedback to the users. But it seems when the step fails, all outputs are empty

mina-kleid-mobile commented 11 months ago

I am currently facing the same issue with test-application. Whenever there is a failing test the steps errors and all the outputs are null.

I can see here that you throw an error without setting any outputs.

if (run_status.run.result != "PASSED") {
        throw("Test run failed after " + i*wait_interval + " seconds with: " + run_status.run.resultCode + ". Timeout is set to " + max_wait*wait_interval);
    }
bwachter commented 11 months ago

I can see here that you throw an error without setting any outputs.

I'll revisit this soon - at the time of writing this I didn't get any meaningful reply to be returned here from the devicefarm API, and therefore just had it fail without any outputs.

mina-kleid-mobile commented 11 months ago

I can see here that you throw an error without setting any outputs.

I'll revisit this soon - at the time of writing this I didn't get any meaningful reply to be returned here from the devicefarm API, and therefore just had it fail without any outputs.

@bwachter Thanks for your reply. In our case the run arn would be very useful. It is already available straight after the schedule run is called. In this case when using the test-application and the run fails we can show a link to the run

bwachter commented 11 months ago

In our case the run arn would be very useful. It is already available straight after the schedule run is called. In this case when using the test-application and the run fails we can show a link to the run

That's a good point - and at minimum I'll add that when I work on this again.

Assumption when building this was that developers calling this typically don't have access to the device farm - so the focus was on pulling enough information to debug errors from the API and output them. Now with more 3rd parties using this action I might have to revisit some of those initial assumptions.

mina-kleid commented 11 months ago

I have added the proposed change here