Closed CesarLeblanc closed 1 year ago
@CesarLeblanc
Hi! Thank you for such a detailed report.
What I see from the error traceback:
key_compression
is calledkey_compression
is a linear layerkey_compression
is zeroThis happened because of the too low value of kv_compression_ratio=0.004
for this number of features. In a nutshell, key_compression
is created as follows:
key_compression = nn.Linear(
num_input_features,
int(kv_compression_ratio * num_input_features), # int(0.004 * 152) = 0
bias=False,
)
The purpose of this is to reduce the number of features to make the attention faster. However, there is no point in reducing the number of features below a certain threshold. This threshold is purely heuristic and should be chosen based on your budget and the downstream performance. In particular, it depends on the number of features.
The best scenario is when you don't need compression (i.e. kv_compression_ratio=None
). If this does not fit into a budget, then choose kv_compression_ratio
from the arange [a, b]
based on your intuition and preference, where a
is the smallest value that still provides good performance in terms of metrics, and b
is the largest value that still fits into you budget.
Does this help?
Dear @Yura52,
Thanks a lot for your answer, it does help!
I had to put such a low value for the kv_compression_ratio
parameter when the training was done on the full dataset (because of my available hardware and time budget), but it's true that it doesn't make a lot of sense to use the same value for this parameter when the first group of features (representing almost 99% of all of the input features) is removed during an ablation study.
So for the moment I simply changed my initialization of the model by modifying three lines:
class FTT(rtdl.FTTransformer):
def __init__(self, n_num_features=None, cat_cardinalities=None, d_token=16, n_blocks=1, attention_n_heads=4, attention_dropout=0.3, attention_initialization='kaiming', attention_normalization='LayerNorm', ffn_d_hidden=16, ffn_dropout=0.1, ffn_activation='ReGLU', ffn_normalization='LayerNorm', residual_dropout=0.0, prenormalization=True, first_prenormalization=False, last_layer_query_idx=[-1], n_tokens=None, kv_compression_ratio=0.004, kv_compression_sharing='headwise', head_activation='ReLU', head_normalization='LayerNorm', d_out=None):
feature_tokenizer = rtdl.FeatureTokenizer(
n_num_features=n_num_features,
cat_cardinalities=cat_cardinalities,
d_token=d_token
)
transformer = rtdl.Transformer(
d_token=d_token,
n_blocks=n_blocks,
attention_n_heads=attention_n_heads,
attention_dropout=attention_dropout,
attention_initialization=attention_initialization,
attention_normalization=attention_normalization,
ffn_d_hidden=ffn_d_hidden,
ffn_dropout=ffn_dropout,
ffn_activation=ffn_activation,
ffn_normalization=ffn_normalization,
residual_dropout=residual_dropout,
prenormalization=prenormalization,
first_prenormalization=first_prenormalization,
last_layer_query_idx=last_layer_query_idx,
n_tokens=None if int(kv_compression_ratio * n_num_features) == 0 else n_num_features + 1, # Modified line
kv_compression_ratio=None if int(kv_compression_ratio * n_num_features) == 0 else kv_compression_ratio, # Modified line
kv_compression_sharing=None if int(kv_compression_ratio * n_num_features) == 0 else "headwise", # Modified line
head_activation=head_activation,
head_normalization=head_normalization,
d_out=d_out
)
super(FTT, self).__init__(feature_tokenizer, transformer)
It's clearly not optimal and can be improved (e.g., by automatically setting the value of kv_compression_ratio with respect to the number of input features and not putting always 0.004), but for the moment it is sufficient as the code is running.
However, I'm having trouble understanding why the code was working when I was running it on a single GPU (1x RTX 2080 Ti) but not on two (2x RTX 2080 Ti). Could you explain this? I understand your answer, but I don't get why the code still runs locally with 152 input features and a kv_compression_ratio
of 0.004 (by "locally" I mean using my own computer that has one GPU).
I should admit I don't have a good explanation for that :)
Feel free to reopen the issue if needed!
Dear maintainers,
I've been using your package for a while now (especially for the FTT model). I've never encountered any trouble, and it helped me boost my performance on a tabular dataset. Recently, I've been doing some ablation studies, i.e., I've removed some input features to check if the performance of the model would decrease (and if yes, at what point). I've discovered that, when using several GPUs (2x RTX 2080 Ti), the training fails when it has a certain number of input features (but not always, it really depends of the number of input features). I'm using
torch.nn.DataParallel
to implement data parallelism, and for some reason related to my framework I don't wish to usetorch.nn.parallel.DistributedDataParallel
.Here's a minimal reproducible example to prove my point:
When my training data (a CSR matrix) has a shape of (803473, 152) (i.e., 803473 samples with each 152 features), this code fails (on multi-GPU). However, if I have a training data of shape (803473, 252) (I just tried a random number), then it works smoothly.
Here are the logs:
Some reminders:
Thanks for your help!