search

lucidrains / x-transformers

A simple but complete full-attention transformer with a set of promising experimental features from various papers
MIT License
4.42k stars 377 forks source link

Multi Input/output transformers #236

Closed RyanKim17920 closed 4 days ago

RyanKim17920 commented 6 months ago

Refer to https://github.com/lucidrains/x-transformers/pull/235, I fixed naming conventions and all merging conflicts. I still haven't tested all the features completely so there may be errors.

lucidrains commented 5 months ago

@RyanKim17920 hey Ryan, this looks like good effort

are you using all the features present in this wrapper? i would suggest just removing everything except for the multi-io portion

RyanKim17920 commented 5 months ago

I believe most of the features should be working properly but I wasn't exactly too sure how to implement the memory-based features so those ones may not implemented well

lucidrains commented 5 months ago

@RyanKim17920 yea, you can just remove the memory-based features, as well as anything that isn't related to what you are working on. just allow the multi-io logic to shine through

could you also add an example execution, in the same style as the other examples in the readme?

lucidrains commented 5 months ago

@RyanKim17920 how well does it work for your project? have you trained anything with it?

RyanKim17920 commented 5 months ago

@RyanKim17920 how well does it work for your project? have you trained anything with it?

Ah, I haven't worked on it in a while so I haven't tested the model system yet. I was trying to generalize the transformers first so that the training process itself would work out smoother

RyanKim17920 commented 4 days ago

While I've worked on the project for a while, I don't think there is much potential ahead due to the terrible quality of training data I have and the extreme specificity and lack of general practicality of the model architecture I proposed to add.