Open GraySerg opened 2 months ago
Hi @GraySerg that's a new one here.
First, let's note what this means: EXECABORT Transaction discarded because of previous errors.
mean's that the transaction failed. And failed from some error, not an error in the command itself but some other kind of system error.
After a bit of googling, this looks like something that can happen if Redis falls into a bad state (e.g. looks like a really popular one is if the Redis Server runs out of memory).
Can you validate the:
INFO MEMORY
)These stats are from a Redis instance where this failure is actively happening?
The oddity here is that EXECABORT Transaction discarded because of previous errors.
means that Redis threw out the entire command out of hand without ever trying to execute, unfortunately this mean's that it discards the actual error's that caused it. There's a few reasons that could happen.
1: There's some major syntactic issue in one of the commands. This seems unlikely given we're doing a JSON set here, and the object is explicitly being serialized to a JSON string. 2: A given command is missing, again this seems unlikely given that you are executing against a Redis Stack Instance, unless you have some non-redis stack instance somewhere in your topography that this could be getting routed to. 3: Some external force (e.g. a memory issue) which Redis would be able to explicitly know about ahead of time. But your memory info says that your system memory is 32 gigs, your max memory setting is 30gigs and your currently used memory is 22mb, so unless this instance is just not representative of your redis deployment then that seems unlikely.
Do you have an exact command/object/timeout this is failing on?
Yes, it from production server. Code looks like
var events = _provider.RedisCollection<EntityToCache>();
var eventItem = await events.FindByIdAsync(message.Key.ToString());
if (eventItem == null)
{
eventItem = new EntityToCache();
MapData(eventData, eventItem);
await events.InsertAsync(eventItem, ttl);//Here exception occured after about 200-400 insert calls
}
else
{
MapData(eventData, eventItem);
await events.UpdateAsync(eventItem);//No exceptions here
}
@GraySerg - do you happen to have a copy of the ttl and the eventData that you're trying to insert when this is happening? (if the eventData
is sensitive, please sanitize)
TLL is 2 days.
var ttl = TimeSpan.FromDays(2);
Class looks like
[Document(StorageType = StorageType.Json, Prefixes = new[] { "EventData" })]
public class EntityToCache
{
[RedisIdField, Indexed]
public int Id { get; set; }
[Indexed]
public bool IsAlive { get; set; }
[Indexed]
public bool IsActive { get; set; }
[Indexed]
public int TopValue { get; set; }
[Indexed]
public DateTime StartTime { get; set; }
public int[] Visibility { get; set; }
[Indexed(CascadeDepth = 1)]
public NameTranslations LeagueNames { get; set; }
[Indexed]
public int TypeId { get; set; }
[Indexed]
public Score Score { get; set; }
[Indexed(CascadeDepth = 1)]
public NameTranslations Team1 { get; set; }
[Indexed(CascadeDepth = 1)]
public NameTranslations Team2 { get; set; }
[Indexed]
public bool HasCoefs { get; set; }
[Indexed]
public bool IsBetAccepted { get; set; }
public int MinBetLimit { get; set; }
[Indexed]
public bool IsVisible { get; set; }
public Dictionary<int, int> Limits { get; set; }
public int MinCount { get; set; }
public int MaxCount { get; set; }
}
public class NameTranslations
{
public NameTranslations()
{
Languages = new Dictionary<int, string>(2);
}
[Indexed]
public Dictionary<int, string> Languages { get; set; }
}
public class Score
{
[Indexed(CascadeDepth = 1)]
public ScoreUnit General { get; set; }
[Indexed(CascadeDepth = 1)]
public ScoreUnit[] Detailed { get; set; }
[Indexed]
public string DetailedString { get; set; }
[Indexed(CascadeDepth = 1)]
public ScoreUnit Points { get; set; }
[Indexed]
public int Innings { get; set; }
[Indexed]
public int Advantage { get; set; }
}
public class ScoreUnit
{
[Indexed]
public int Team1 { get; set; }
[Indexed]
public int Team2 { get; set; }
}
Hi @GraySerg - I am unable to reproduce this error on my end with the similar objects and timeout, with much heavier loads than what you're quoting. Can you able to share a reproduction of this issue outside of your production environment?
IMO the likeliest cause is still environmental (there just isn't any reason the commands you quoted should be failing in this way)
It happens only in Redis cluster. If Redis only on one node it works fine.
Oh interesting. . . Curious, how many shards are in your Redis Cluster? Is it possible one of those shards doesn't have Redis Stack Running but rather some vanilla version of Redis without the module support? Because a missing command would cause the above error.
We have 3 shards by default, and all 3 equals Redis Stack.
Hi @slorello89 We found the problem: when we send Json.Set commend - redis cluster returns HashSlot MOVER on each request. It happens because in RedisOM used string key. But if cast string to RedisKey all requests will be executing success without MOVED, because StackExchange.Redis can calculate right HashSlot for RedisKey type. Can you change it behavior in RedisOM library?
On production server frequently occurs that error. Collection contains about 6000 entities.
StackExchange.Redis.RedisServerException: EXECABORT Transaction discarded because of previous errors. at Redis.OM.RedisConnection.ExecuteInTransactionAsync(Tuple2[] commandArgsTuples) at Redis.OM.RedisCommands.JsonSetAsync(IRedisConnection connection, String key, String path, String json, TimeSpan timeSpan) at Redis.OM.RedisCommands.JsonSetAsync(IRedisConnection connection, String key, String path, Object obj, TimeSpan timeSpan) at Redis.OM.RedisCommands.SetAsync(IRedisConnection connection, Object obj, TimeSpan timeSpan) at Redis.OM.Searching.RedisCollection1.InsertAsync(T item, TimeSpan timeSpan)
No information about "previous errors". Update entites in collection works fine