-
Ok, this is more of a question about transformers in general and not about Llama being different from the standard transformer architecture: why is there no activation on the assembled values, just be…
jxtps updated
12 months ago
-
In [Unsloth](https://github.com/unslothai/unsloth/blob/main/unsloth/__init__.py#L53-L54):
```python
# Reduce VRAM usage by reducing fragmentation
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable…
-
Here is what I am getting (see below)
FP8 slower than FP16
for FP16, multiples of 16 make things slower than multiple of 8
Am I missing something ?
Batch_size_multiple 16 // Seqlen multi…
-
Im BibTeX-Export werden Enrichment-Felder gemappt, die nicht Teil der OPUS-Standardauslieferung sind und, von zwei Ausnahmen abgesehen, in den gehosteten Instanzen nicht verwendet werden, Diese sind:
…
-
bsz: 158
num_dev: 6
gpu0_bsz: 1
bsz_unit: 31
chunk_sizes: [1, 32, 32, 31, 31, 31]
len(inputs): 6
self.device_ids[:len(inputs)] [0, 1, 2, 3, 4, 5]
replicas: 6
+-----------------------…
-
https://github.com/liyi01827/noisy-contrastive/blob/03f48ca3028f4a6fc54b6faaf05ad811b4055381/main.py#L216
hi, I have some question, about contrast_logits=2+contrast_1. What is the meaning of 2?
co…
-
Hello, nice work on the training code! Thank you for sharing this code.
I have a question about your image conditioned classifier guidance in your code.
```python
if args.conditioning_dropout_prob …
-
hi i have observed significant performance degradation in multi-gpu with grad accum > 1 setting.
I'm sorry not to upload profiling code (will be uploaded soon) but i tested 7B scale LLM using same si…
-
Hey, Can Supervised Contrastive Loss be implemented for Semantic Segmentation where the output of the model is of the shape;
**features = [bsz, feat_dim, h, w]
target = [bsz, h, w]
target_embedding…
-
In each iteration, the state of the `self.q_deivce` will be changed. Why is the changed `self.q_device` in the previous iteration used in the next iteration here?
```
class QuanvolutionFilter(tq.Qua…