Closed chitalian closed 1 year ago
Seems as this mirror is down, which is currently the official mirror of the NeoX 20B weights. I guess we should get our own, given that we trained the model and all.
@chitalian you can change the address from https://mystic.the-eye.eu/public/AI/models/GPT-NeoX-20B/slim_weights/
to https://the-eye.eu/public/AI/models/GPT-NeoX-20B/slim_weights/
in the download job script and it should work.
@chitalian you can change the address from
https://mystic.the-eye.eu/public/AI/models/GPT-NeoX-20B/slim_weights/
tohttps://the-eye.eu/public/AI/models/GPT-NeoX-20B/slim_weights/
in the download job script and it should work.
That seemed to do the trick!! thanks for the quick response
Want me to make a PR to fix this?
Please!!
@salanki Really strange actually...
I got a lot further but I left the logs open when i was retrying and got this
FINISHED --2022-11-08 19:40:25--
Total wall clock time: 19m 31s
Downloaded: 108 files, 38G in 19m 3s (34.3 MB/s)
Converting from 2 to 1 GPUs
Strategy: group 2 source gpu(s) into 1 out gpu(s).
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/mnt/pvc/gpt-neox/FasterTransformer/examples/pytorch/gptneox/utils/eleutherai_gpt_neox_convert.py", line 163, in handle_layer
tensor.tofile(save_dir / ("model." + output_name + ".bin"))
OSError: problem writing element 131747840 to file
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/mnt/pvc/gpt-neox/FasterTransformer/examples/pytorch/gptneox/utils/eleutherai_gpt_neox_convert.py", line 305, in <module>
convert_checkpoint(args)
File "/mnt/pvc/gpt-neox/FasterTransformer/examples/pytorch/gptneox/utils/eleutherai_gpt_neox_convert.py", line 282, in convert_checkpoint
pool.starmap(handle_layer, handle_layer_args)
File "/usr/lib/python3.8/multiprocessing/pool.py", line 372, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
raise self._value
OSError: problem writing element 131747840 to file
mv: target '/mnt/pvc/triton-model-store/fastertransformer/1/' is not a directory
/bin/bash: line 245: echo: write error: Disk quota exceeded
also on my dashboard the storage instance looks empty 🤔
Can you try to increase the volume? It shows as 0% because it is not currently mounted, the text below the 0% is trying to explain that.
fixed with #100
This is still an issue. Will open a new issue.
Please do, add as much detail as you can.
When going through this tutorial: https://docs.coreweave.com/compass/examples/triton-inference-server-fastertransformer
At this step
I am running into this error: