Autofarm: Cache galaxyinfo, persist database

[Deenayd]

Hello,

It's a long one ;)

It would be nice to implement caching of galaxyInfo received from the server. It would allow more aggressive sending of farming attacks without spamming galaxy refreshes.

Cache (or it's entry at least) should be cleared on any error with sending ships to the destination. In case an error is caused by planet (or moon) got removed.

It would also be nice to persist galaxy cache and farm targets.

And it would be nice to implement deadline for scanning targets.

For me it looks like this:

declare: static volatile Dictionary<int, GalaxyInfo> galaxyCache = new();

At start of AutoFarm - prepare:

            string farmTargetsFileName = new($"{Helpers.logPath}/farmTargets.json");
            string galaxyCacheFileName = new($"{Helpers.logPath}/galaxyCache.json");
            var rng = new Random();
            DateTime? ScanningDeadTime = null;

After checking if there are enough slots - read persisted data and change pruning old data

                    if (freeSlots > slotsToLeaveFree) {
                        try {
                            if (farmTargets.Count == 0) {
                                try {
                                    string farmTargetsStr = File.ReadAllText(farmTargetsFileName);
                                    farmTargets = Newtonsoft.Json.JsonConvert.DeserializeObject<List<FarmTarget>>(farmTargetsStr);
                                } catch (Exception) {
                                }
                            }
                            if (galaxyCache.Count == 0) {
                                try {
                                    string galaxyCacheStr = File.ReadAllText(galaxyCacheFileName);
                                    galaxyCache = Newtonsoft.Json.JsonConvert.DeserializeObject<Dictionary<int, GalaxyInfo>>(galaxyCacheStr);
                                } catch (Exception) {
                                }
                            }
                            // Prune all reports older than KeepReportFor and all reports of state AttackSent: information no longer actual.
                            var newTime = GetDateTime();
                            var removeReports = farmTargets.Where(t => t.State == FarmState.AttackSent || t.State == FarmState.ProbesSent || (t.Report != null && DateTime.Compare(t.Report.Date.AddMinutes((double) settings.AutoFarm.KeepReportFor), GetDateTime()) < 0)).ToList();
                            foreach (var remove in removeReports) {
                                var updateReport = remove;
                                updateReport.State = FarmState.ProbesPending;
                                updateReport.Report = null;
                                farmTargets.Remove(remove);
//                              farmTargets.Add(updateReport);
                            }

At scanning loop - check deadline:

                            foreach (var range in settings.AutoFarm.ScanRange) {
                                if (Helpers.IsSettingSet(settings.AutoFarm.TargetsProbedBeforeAttack) && settings.AutoFarm.TargetsProbedBeforeAttack != 0 && numProbed >= (int) settings.AutoFarm.TargetsProbedBeforeAttack) break;
                                if ((ScanningDeadTime != null) && (DateTime.Now > ScanningDeadTime)) break;

                                int galaxy      = (int) range.Galaxy;
                                int startSystem = (int) range.StartSystem;
                                int endSystem   = (int) range.EndSystem;

A little lower - check deadline again:

                                // Loop from start to end system.
                                for (var system = startSystem; system <= endSystem; system++) {
                                    if (Helpers.IsSettingSet(settings.AutoFarm.TargetsProbedBeforeAttack) && settings.AutoFarm.TargetsProbedBeforeAttack != 0 && numProbed >= (int) settings.AutoFarm.TargetsProbedBeforeAttack) break;
                                    if ((ScanningDeadTime != null) && (DateTime.Now > ScanningDeadTime)) break;

                                    // Check excluded system.

A little bit lower - cache galaxyInfo. I have decided to randomly refresh it sometimes. You can do that or not

                                    GalaxyInfo galaxyInfo;
                                    int galaxyIndex = galaxy * 1000 + system;
                                    {
                                        if (rng.NextDouble() < 0.02) {
                                            galaxyCache.Remove(galaxyIndex);
                                        }
                                        if (galaxyCache.TryGetValue(galaxyIndex, out galaxyInfo)) {
                                        } else {
                                            galaxyInfo = ogamedService.GetGalaxyInfo(galaxy, system);
                                            galaxyCache.Add(galaxyIndex, galaxyInfo);
                                        }
                                    }
                                    var planets = galaxyInfo.Planets.Where(p => p != null && p.Inactive && !p.Administrator && !p.Banned && !p.Vacation && p.Coordinate.Type == Celestials.Planet);
                                    List<Celestial> scannedTargets = planets.Cast<Celestial>().ToList();

Later inside loop of sending probes we need to update free slots in case of an error. Else we risk of ugly loop of failed attempts if another task used our last fleet slot:

                                            if (celestialProbes[closest.ID] >= neededProbes) {
                                                Ships ships = new();
                                                ships.Add(Buildables.EspionageProbe, neededProbes);

                                                Helpers.WriteLog(LogType.Info, LogSender.AutoFarm, $"Spying {target.ToString()} from {closest.ToString()} with {neededProbes} probes.");

                                                slots = UpdateSlots();
                                                // in case another task used our fleet slot we need to break a loop of failed spy attempts
                                                freeSlots = slots.Free;
                                                var fleetId = SendFleet(closest, ships, target.Celestial.Coordinate, Missions.Spy, Speeds.HundredPercent);
                                                if (fleetId > 0) {

Just a little bit lower - chose a deadline after first successful spy. Could be taken from settings probably:

                                                if (fleetId > 0) {
                                                    freeSlots--;
                                                    numProbed++;
                                                    celestialProbes[closest.ID] -= neededProbes;

                                                    if (ScanningDeadTime == null) {
                                                        ScanningDeadTime = DateTime.Now.AddMinutes(10); // could be settings parameter
                                                    }

                                                    if (target.State == FarmState.ProbesRequired || target.State == FarmState.FailedProbesRequired)
                                                        break;

Remove an entry from galaxyCache in case of trouble:

                                                } else if (fleetId == -1) {
                                                    stop = true;
                                                    return;
                                                }
                                                else {
                                                    // remove from galaxy cache, error might mean the planet has dissappeared
                                                    galaxyCache.Remove(galaxyIndex);
                                                    break;
                                                }

Oh, fix calculation of time required to wait for spy reports. We need to wait for all the spy reports, not for returning of first spy probe. There might be more probes still heading to their destinations. I have choced a fixed timeout to avoid waiting for any slow espionage fleet saves:

                        // Wait for all espionage fleets to return.
                        fleets = UpdateFleets();
                        var celestialEspionages = Helpers.GetMissionsInProgress(Missions.Spy, fleets).Where(e => (e.ArriveIn >= 0) && (e.ArriveIn < 4 * 60));
                        Fleet lastSpied = (celestialEspionages.Any())
                            ? celestialEspionages.OrderByDescending(e => e.ArriveIn).First()
                            : null;
                        //Fleet firstReturning = Helpers.GetFirstReturningEspionage(fleets);
                        if (lastSpied != null) {
                            int interval = (int) ((1000 * lastSpied.ArriveIn) + Helpers.CalcRandomInterval(IntervalType.AFewSeconds));
                            Helpers.WriteLog(LogType.Info, LogSender.AutoFarm, $"Waiting for probes to spy...");
                            Thread.Sleep(interval);
                        }

And finally store galaxyinfo and galaxycache to files:

            } finally {
                Helpers.WriteLog(LogType.Info, LogSender.AutoFarm, $"Attacked targets: {farmTargets.Where(t => t.State == FarmState.AttackSent).Count()}");
                {
                    if (farmTargets.Count > 0) {
                        try {
                            string farmTargetsStr = Newtonsoft.Json.JsonConvert.SerializeObject(farmTargets, Newtonsoft.Json.Formatting.Indented);
                            File.WriteAllText(farmTargetsFileName, farmTargetsStr);
                        } catch (Exception) {
                        }
                    }
                    if (galaxyCache.Count > 0) {
                        try {
                            string galaxyCacheStr = Newtonsoft.Json.JsonConvert.SerializeObject(galaxyCache, Newtonsoft.Json.Formatting.Indented);
                            File.WriteAllText(galaxyCacheFileName, galaxyCacheStr);
                        } catch (Exception) {
                        }
                    }
                }
                if (!isSleeping) {

Sorry for not generating a patch, but I have too many changes made to the files and it's not easy to split them to different patches now :(

If data are persistent, than it's possible to add a lot more logic. But more about that another time.

[ogame-tbot]

i like this! why don't you get into discord server so we can discuss better?

[Deenayd]

Ok. I will try to. Tomorrow.