search

jianjieluo / SCD-Net

[CVPR23] A cascaded diffusion captioning model with a novel semantic-conditional diffusion process that upgrades conventional diffusion model with additional semantic prior.
https://arxiv.org/abs/2212.03099
Other
57 stars 5 forks source link

Question about Generating Repeated Tokens while Inferencing #6

Closed AlonzoLeeeooo closed 5 months ago

AlonzoLeeeooo commented 1 year ago

Hi @jianjieluo ,

Thanks for the amazing work! I am working on transferring the architecture of SCD-Net upon medical radiology reports generation. But I figure out that the trained model tends to generate repeated tokens during testing. Have you ever encountered this kind of situation before?

The training losses and visualized results are shown as below.

image image

Note that the grey curve in the first figure is the training loss on a small medical dataset (only about 5,000 images). The yellow curve is the one that produces repeated tokens.

Could you please help me out with this? Thanks you in advance for replying from your busy schedule.

Best,

mbrz97 commented 5 months ago

Hi @AlonzoLeeeooo,

Did you reach any promising results running this on your medical dataset? I'm also working applying this model to a medical dataset. I'd really appreciate it if you could let me know how it went.

Thanks!

AlonzoLeeeooo commented 5 months ago

Hi @AlonzoLeeeooo,

Did you reach any promising results running this on your medical dataset? I'm also working applying this model to a medical dataset. I'd really appreciate it if you could let me know how it went.

Thanks!

Hi @mbrz97 ,

You can try this repo, which is an implementation for radiology report generation (RRG).

mbrz97 commented 5 months ago

@AlonzoLeeeooo Thank you! This is incredibly helpful.

I'm also curious to know if you were able to resolve the repeating token problem mentioned in your original post.

AlonzoLeeeooo commented 5 months ago

@AlonzoLeeeooo Thank you! This is incredibly helpful.

I'm also curious to know if you were able to resolve the repeating token problem mentioned in your original post.

I've solved the token repeated problem by using another repo, which is originated from this repo and adapted for radiology report generation on IU X-Ray and MIMIC-CXR. You can try training SCD-Net with its codebase.