I would like to implement LongNet for a project that is inputting numerical data into a transformer, to predict numerical data. However, for my data there are connections between each data point in the input sequence over the entire range of the input.
This means that segment lengths and dilation rates chosen by a human user might not make sense. So I wanted to ask if there is a way of learning the best segment lengths and dilation rates, based on the connections in the input sequence that the model might find?
Hi there,
I would like to implement LongNet for a project that is inputting numerical data into a transformer, to predict numerical data. However, for my data there are connections between each data point in the input sequence over the entire range of the input.
This means that segment lengths and dilation rates chosen by a human user might not make sense. So I wanted to ask if there is a way of learning the best segment lengths and dilation rates, based on the connections in the input sequence that the model might find?
Many thanks.