Closed amit13k closed 9 months ago
lapp0
commented
9 months ago Related: https://github.com/outlines-dev/outlines/issues/656
Looks like you're on an outdated version of outlines: outlines@git+https://github.com/lapp0/outlines@e99d92d
Could you try pip install --upgrade outlines?
amit13k
commented
9 months ago Related: #656
Looks like you're on an outdated version of outlines:
outlines@git+https://github.com/lapp0/outlines@e99d92dCould you try
pip install --upgrade outlines?
Thanks for the reply. I uninstalled outlines and ran pip install --upgrade outlines (which installed version 0.0.32), and the issue still exists. Creating and using the generator in a different thread leads to the same error.
lapp0
commented
9 months ago Thanks for checking, sorry it's not still working. I'll take a look soon.
lapp0
commented
9 months ago Could you please try using a fresh venv? I cannot reproduce and it appears you're using a local exllamav2 (exllamav2 @ file:///home/amit/repos/outlines/exllamav2)
If the issue persists with a fresh venv, please provide your new pip3 freeze, along with nvidia-smi.
Script:
import outlines
import threading
# git lfs install
# git clone https://huggingface.co/MaziyarPanahi/miqu-1-70b-sf-GPTQ models/llm/miqu
model = outlines.models.exl2(model_name="models/llm/miqu", model_kwargs={
"num_experts_per_token": 1,
"gpu_split": "18,24",
}, device="cuda")
def task():
s_greedy = outlines.samplers.greedy()
s_multinomial = outlines.samplers.multinomial()
for sampler in [s_greedy, s_multinomial]:
print("generating with", sampler)
generator = outlines.generate.text(model, sampler)
output = generator("What is gravity?")
print(output)
thread = threading.Thread(target=task)
thread.start()Output:
generating with <outlines.samplers.GreedySampler object at 0x7f289c11c640>
Gravity is a force that pulls two objects towards each other. It is a fundamental force of nature, which means that it is one of the basic forces that govern the behavior of matter and energy in the universe. Gravity is what keeps planets in orbit around stars, and it is what keeps objects on the surface of a planet.
Gravity is described by the theory of general relativity, which was developed by Albert Einstein in 1915. According to this theory, gravity is not a force that acts at a distance, but rather a curvature of space-time caused by the presence of mass or energy. This means that objects move in response to the curvature of space-time, rather than being pulled by a force.
The strength of gravity between two objects depends on their masses and the distance between them. The greater the masses of the objects, the stronger the gravitational force between them. The closer the objects are to each other, the stronger the gravitational force.
Gravity is a very weak force compared to other fundamental forces, such as the strong and weak nuclear forces that govern the behavior of subatomic particles. However, because of the large masses of objects in the universe, gravity has a very significant effect on the motion of celestial bodies and the structure of the universe as a whole.
generating with <outlines.samplers.MultinomialSampler object at 0x7f289c11cac0>
How does it affect motion in our solar system and beyond? Let's explore an intriguing relationship between mass, energy, and gravitation.
Gravity is a fundamental force of nature that describes the attractive interaction between mass or energy. It's responsible for holding planets in orbit around stars, maintaining structures like galaxies, and even keeping our feet on the ground.
The universal law of gravitation, proposed by Sir Isaac Newton in 1687, explains how the strength of gravitational attraction between two objects depends on their masses and the distance between them:
F = G * (m1 * m2) / r_
Where F is the force of gravity, G is the gravitational constant, m1 and m2 are the masses of the two objects, and r is the distance between them.
In our solar system, this law explains why planets move in elliptical orbits around the Sun. The Sun, being much more massive than any planet, exerts a dominant gravitational pull that keeps the planets in their orbits. This idea was further refined by Albert Einstein's theory of General Relativity, which describes gravity as a curvature of spacetime caused by mass and energy.
Outside our solar system, gravity plays a crucial role in the formation and evolution of stars, galaxies, and even cosmic structures like dark matter halos. Stars, formed from collapsing clouds of gas and dust, burn through their nuclear fuel, eventually exploding as supernovae, releasing vast amounts of energy and creating new elements. These stars then form binary systems, units of two stars orbiting each other, or become part of larger systems where their gravity influences the motion of planets and other celestial bodies.
Galaxies, which contain millions or billions of stars, are held together by their mutual gravitational attraction. The presence of dark matter, an unknown substance that does not emit light but has gravitational effects, is believed to provide the necessary mass to keep galaxies stable against the outwards forces of their own stars.
Gravity's influence extends far beyond our solar system, playing a role in the large-scale structure of the universe. Studies suggest that more than 95% of the universe's matter is composed of dark matter and dark energy. While the nature of these mysterious substances remains unknown, our understanding of gravity continues to be a cornerstone in explaining their effects on the cosmic scale.
Gravity, an intriguing force that has shaped our understanding of the universe, continues to be a topic of fascination and research for scientists, astronomers, and curious individuals alike. As we continue to probe deeper into the secrets of the universe, our knowledge of gravity will undoubtedly evolve, providing new insights into the fundamental structure of reality.
Thanks for trying to reproduce the issue. I tried to log the device info before the line where the error is happening in samplers.py.
print(f"sequence_weights.device: {sequence_weights.device}, logprobs.device: {logprobs.device}, next_token_ids.device: {logprobs.device}")
weights = sequence_weights + torch.gather(logprobs, 1, next_token_ids).squeeze()When trying to run the generation in a new thread, the log indicatd that sequence_weights was on cuda:0,
sequence_weights.device: cuda:0, logprobs.device: cuda:1, next_token_ids.device: cuda:1
and when running in the main thread or using device="cuda:1", everything was on cuda:1,
sequence_weights.device: cuda:1, logprobs.device: cuda:1, next_token_ids.device: cuda:1
Wondering why creating a new thread changes the device for sequence_weights. However, I was able to fix the issue by changing device="cuda" to device="cuda:1" when creating the model.
rlouf
commented
9 months ago Glad this workaround made it work! Would you mind printing the devices of weights, prompt_token_ids and attention_masks here? I suspect torch assigns devices more or less at random when only cuda is specified. In this case we will need to update the initialization code to make sure attention_masks and sequences_weights are on the same device as prompt_token_ids.
This is a small change, happy to review a PR if you feel like contributing.
Describe the issue as clearly as possible:
I was attempting to expose outlines features with a Flask server and encountered the following error:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!Upon debugging, it seems this error arises if I create
generator = outlines.generate.text(model)in a different thread than where the model is instantiated (which is what happens in Flask). When I run all the code in the same thread, it works. Also in Flask, I am acquiring a lock before calling any of the outlines functions.Additionally, this error doesn't seem to occur when using smaller models that fit completely on 1 GPU. I wish it were possible to work with larger models. Any suggestions to fix this issue would be appreciated. Thanks.
Steps/code to reproduce the bug:
Expected result:
Error message:
Outlines/Python version information:
Version information
Context for the issue:
No response