Epoch Settings in Tangram and Cell2Location: Potential Impact on Model Performance

[Renpeng007]

Hi,

I noticed that in your implementation of Tangram, you use the following code for training epochs:

ad_map = tg.map_cells_to_space(adata_sc,adata_st, mode="cells", density_prior='rna_count_based', num_epochs=100, device="cpu")

The num_epochs here is set to 100, which is significantly smaller than the default value of 1000 used in the official tutorial (as seen here: Tangram official tutorial). I am concerned that the model may not have reached convergence, which could lead to unreliable cell annotation results from Tangram.

I noticed a similar situation with the cell2location implementation. In your code, you're using 250 epochs for training during the mapping step:

` mod.train(max_epochs=250,

train using full data (batch_size=None)

      batch_size=500,
      # use all data points in training because
      # we need to estimate cell abundance at all locations
      train_size=1
      #,use_gpu=True
      )`

However, the official documentation recommends using max_epochs=30000 for training (as mentioned here: cell2location tutorial), which is substantially larger than what you're using.

I believe these differences in training parameters could significantly affect the performance of the models and the validity of the results. Could you please clarify the reasoning behind the choice of these reduced epochs, or consider rerunning the experiments with the recommended settings to ensure the results are more reliable?

Thanks in advance for your consideration. Looking forward to your response!

[qyTao185]

Dear RenPeng,

Thank you for your thoughtful email and for raising such an important point regarding the training epochs used in Tangram and cell2location. I truly appreciate your attention to detail, and I’d be happy to provide some clarification.

You are absolutely right that the default number of epochs in the official tutorials is set to 1000 or higher. In our implementation, we intentionally reduced the number of epochs primarily to enhance efficiency during the early stages of pipeline testing and validation. This allowed us to quickly fine-tune other hyperparameters before committing to longer training times. While we agree that using more epochs can increase the likelihood of full model convergence, especially when aiming for reliable cell annotations, we have also observed that the optimal number of epochs can depend heavily on the quality and nature of the data. In our tests, for some datasets, increasing the number of epochs did not necessarily improve results and, in some cases, even led to overfitting or degraded performance.

That said, we acknowledge that there isn’t a one-size-fits-all parameter that works perfectly across all datasets. We always strive to balance computational efficiency with result quality.

We greatly appreciate your valuable feedback. Your input helps us ensure that we continue to follow best practices, and we are grateful for your understanding.

Best regards

At 2024-10-12 02:32:18, "Renpeng007" @.***> wrote:

Hi,

I noticed that in your implementation of Tangram, you use the following code for training epochs:

ad_map = tg.map_cells_to_space(adata_sc,adata_st, mode="cells", density_prior='rna_count_based', num_epochs=100, device="cpu")

The num_epochs here is set to 100, which is significantly smaller than the default value of 1000 used in the official tutorial (as seen here: Tangram official tutorial). I am concerned that the model may not have reached convergence, which could lead to unreliable cell annotation results from Tangram.

I noticed a similar situation with the cell2location implementation. In your code, you're using 250 epochs for training during the mapping step:

mod.train(max_epochs=250, # train using full data (batch_size=None) batch_size=500, # use all data points in training because # we need to estimate cell abundance at all locations train_size=1 #,use_gpu=True )

However, the official documentation recommends using max_epochs=30000 for training (as mentioned here: cell2location tutorial), which is substantially larger than what you're using.

I believe these differences in training parameters could significantly affect the performance of the models and the validity of the results. Could you please clarify the reasoning behind the choice of these reduced epochs, or consider rerunning the experiments with the recommended settings to ensure the results are more reliable?

Thanks in advance for your consideration. Looking forward to your response!

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[Renpeng007]

Hi Quyuan,

Thank you again for your thoughtful response.

I find the potential overfitting phenomenon you mentioned particularly interesting. I had always assumed that Tangram's design, especially with its built-in regularization part, would prevent overfitting. Do you have any experimental results confirming this phenomenon? If confirmed, I believe this would be a valuable insight and could add significant depth to the evaluation of the tools.

I also understand the need to reduce training time, especially considering you were running the experimentsonly only on CPUs. However, the extent of the reduction in training epochs seems quite large. I wonder if you have examined the loss curves to see whether the models truly converged. Additionally, have you evaluated whether this reduction in training time might have negatively impacted the results? I believe this is worth further investigation, as it could have important implications for the reliability of the findings.

Thank you again for your consideration, and I look forward to hearing your thoughts on these points.