Adding GPUs to jobs without `remote`

[aittalam]

Jobs which do not rely on ray.remote, such as lm-harness, run without GPUs by default.

If we want to add GPUs to our jobs, we need to manually add gpus to ray.submit (see here), for instance when it is called in the example notebooks.

This is ok, I just wanted to raise this to make sure we describe this behavior in the documentation. The reason is that lm-harness jobs break if e.g. we run an evaluation on a quantized model without using a GPU.

[sfriedowitz]

A note on this can be added to the documentation in the bottom of the README##Ray job submission section.

[veekaybee]

Just got this issue as well replicating a run from scratch, will open a PR for it

[veekaybee]

Resolved with the above PR ^