Coordinating the next release

[pjbroad]

We have a full client update coming that includes maps updates (from @feeltheburn) and a server update (from @raduprv). This will be the next major version bump. We've been holding off major changes to the client code so but have the exciting prospect of #132 (from @gvissers) so we need to decide whether to merge that so we can test fully before the release. I'm in favour of merging #132 but what do others think? Are there other changes should we consider?

[raduprv]

Before we do this, we must compile the Android version for 64b, or else Google won't allow it on google play. And the maps were not tested at all on the test server.

[gvissers]

As I said elsewhere, whether we merge the encryption code should IMO depend on how much time we have for the next release. If there's still a month or two left, merging now will give us enough time to get this tested properly and work out the remaining kinks, I think. There's probably still some work to be done, as I have tested this only lightly on Windows, and not at all on MacOS or Android (and FreeBSD? does anyone use that? Maybe I should set up a VM). Worst case, compiling without USE_SSL should give you the original network code, and the other new code from #132 will then never be used.

[pjbroad]

Before we do this, we must compile the Android version for 64b, or else Google won't allow it on google play. And the maps were not tested at all on the test server.

Would this be a version compiled from the code base I've been working on (the android branch)? If so, a 64b version is already included and can be made the only version included if that is what's required. On the maps, didn't mean to imply they had been tested and understand form Burn they will be available to test soon.

As I said elsewhere, whether we merge the encryption code should IMO depend on how much time we have for the next release. If there's still a month or two left, merging now will give us enough time to get this tested properly and work out the remaining kinks, I think. There's probably still some work to be done, as I have tested this only lightly on Windows, and not at all on MacOS or Android (and FreeBSD? does anyone use that? Maybe I should set up a VM). Worst case, compiling without USE_SSL should give you the original network code, and the other new code from #132 will then never be used.

Are you building the windows version using the msys environment setup I also use? I recently installed FreeBSD in a VM so that I could get the cmake file work for BSD; there's a example cmake line in the CMakeList.txt comments. The latest main branch works fine on FreeBSD and I could easily try the encryption version on the same VM if you like which I see elsewhere you are trying FreeBSD yourself.

[raduprv]

Would this be a version compiled from the code base I've been working on (the android branch)? If so, a 64b version is already included and can be made the only version included if that is what's required.

I am not sure exactly, I have to check with google. Is there a way to pack both 32b and 64b in the same apk? Because a lot of older phones are 32b only.

On the maps, didn't mean to imply they had been tested and understand form Burn they will be available to test soon.

Yes, I know, I just stated that before we can think of an update we need to test the maps properly. They are on the test server now.

[feeltheburn]

On the map end, I've been doing some thorough mapwalk checking of every change, still working on that. Also putting together a bundle of new elms, updated tab maps, and a list of changes for people with a separate client set up for Test server so they can check them out as well. (Dropping that on the general population for testing in the past has only led to people screwing up their main client...)

Thanks to testing scripts I created (Learner uses them on OL maps and helped with a few tweaks of them), I was able to test and fix a lot of stuff that map makers in the past would not have caught. You can see a compiled result of tests on one map here for example, it also explains what the marking is: https://el-db.com/tempstuff/map6nf_cave-test-results.png That's actually a combined image. My script creates a PSD file and each of those numbered tests are on separate layers so they can be looked at individually. The scripts actually test a lot of stuff not seen there as well.

So it's mostly just a visual thing now, and human testing of all use_areas and teleports to ensure they go to the right place, and if new/changed places are too cluttered for play, or any timed/walk damage is too much/little. Things that can't always be determined.

Thus far the fixes I've found that need to be done are minimal, nothing game-breaking. Should give testing a week or two at least, with at least one update with fixes of issues I've found coming as soon as I feel the changes are fully tested and I've caught everything I can.

My opinion is we could aim for an August release, but if we aim for September we could give more time for the above-mentioned issues including the possibility of use_ssl to be implemented, and I could get some additional graphics work done (making more unique inventory item images for stuff sharing the same image like I did last time for example). Given how long it's been, I don't think September would be unreasonable.

I'm done with the maps though, other than fixing stuff found in test of course. So my part including testing could be done by August if that's preferred.

[gvissers]

Are you building the windows version using the msys environment setup I also use?

Yes. Thanks for setting up detailed instructions, BTW, they were really helpful.

I recently installed FreeBSD in a VM so that I could get the cmake file work for BSD; there's a example cmake line in the CMakeList.txt comments.

I did the same here. I don't know what I did wrong the first time around, ended up struggling with finding Cal3D and building and patching my own from source. But it works fine (after a small #include fix) with system Cal3D and the line suggested in CMakeLists.txt, so I must have messed something up the first time. Could be that I was working from an old branch. The SSL code also works without further adaptations, so that all seems in order. (it's slow though in the VM, I get max 8FPS inside a building, 2 FPS outside, I'll have to see if I can tweak the VM settings to help that. Even windows is better).

I'll be on vacation in the first two weeks of August, so if we include the encryption I'd prefer to release after that, i.e. second half of August/September. I'm in favor of merging it, though (totally unbiased of course). Perhaps after we check if it works on Android, but I don't know how to build for that.

[pjbroad]

Perhaps after we check if it works on Android, but I don't know how to build for that.

There's an Android branch which I just updated. You could try using these build instructions. I've got to update the assets download for 1.9.5p9 though the previous version should work OK.

I am not sure exactly, I have to check with google. Is there a way to pack both 32b and 64b in the same apk? Because a lot of older phones are 32b only.

You can specify which binary to build, I've been including arm64-v8a and armeabi-v7a. The apk includes both builds but I do not know how they are used nor how to tell which arch my devices are.

[gvissers]

There's an Android branch which I just updated. You could try using these build instructions.

Gaahh... how many goats did you have to sacrifice before you had all that working!? Thank goodness you already worked this out, because I never would have.

I set up a clean ubuntu install in a virtualbox VM, and followed the rest of your instructions. The ssl branch merges cleanly, and builds with a small patch without USE_SSL (and as I'm typing this I see there's two typos in the error message we might want to correct as well):

diff --git a/events.c b/events.c
index 655870f2..43fbeaad 100644
--- a/events.c
+++ b/events.c
@@ -257,7 +257,7 @@ int HandleEvent (SDL_Event *event)
 #ifdef ANDROID
                case SDL_APP_DIDENTERFOREGROUND:
                        SDL_Log("App returned to forground");
-                       if (disconnected && !locked_to_console)
+                       if (is_disconnected() && !locked_to_console)
                        {
                                SDL_Log("Reconnectecing after return to forground");
                                connect_to_server();

Unfortunately I have as of yet not been able to get OpenSSL to build in your build environment. I'm sure it can be done, but I can't seem to get the paths right. So as of yet, I haven't tested the new network code on android yet.

[feeltheburn]

So would an aim for a "first half of September" for a release be reasonable? Latest end of September if unforeseen issues arise or more time is needed for the SSL thing?

Update on my side:

- Maps are looking good. Several have been on and tested the new map areas. Only a handful of minor issues found, all text (like text that gets shown when you use a door or such), so pretty good to go. I'll send a minor update pack to radu in a week or so once others have had more time to explore the new areas and ensure no further issues.

- New Images for Inventory. With thoughts of a late august release earliest (since gvissers will be on vacation the first part of august), I went ahead and started a load of work on inventory images. Specifically, doing something that's long been requested, the ability to differentiate graphically between a weapon/armor and its second-hand/used/degraded/pre-owned counterpart, as they currently look the same in inventory. Working on both those and the updates needed on the server's items.def file to change the graphics for the degraded items. (I showed a few samples of this in @@6, and got cheers. It's been wanted for as long as I've been playing.)

- Possible to do with a September release: Also looking at unique images for the "special" forms of items (ex: Tit cuisses of cooldown removal vs. regular Tit cuisses) as they also share the same images. Basically if it's not a book (too many of those), if I can make a unique image for it, I'm going to try to make one.

[gvissers]

I finally managed to build a client with SSL support on Android, and what a fun process that was :grimacing: :sweat: . Anyway, I have sent out pull requests for the affected repositories (Eternal-Lands/android branch, and el-build-methods). I have verified that it works on my phone, both encrypted and unencrypted, but it has only been tested on the one device (I believe 64bit, but I'd have to check to be sure).

Since it seems to work on all platforms I've tried (Linux, Windows, FreeBSD, Android), I will merge the SSL branch soonish (tomorrow, maybe wednesday) if there are no objections.

@feeltheburn , I've been meaning to test the maps, but haven't found time yet.

[feeltheburn]

That's okay grum, I think we've got enough testers, including explorers that know the maps well like angler. But there's still time so send input if you have time later and see any issues.

[gvissers]

I have merged #132 and enabled USE_SSL by default so that the code can get wider testing. Things should work as they were on unencrypted connections. Encrypted connections can be tested on Learner's proxies, just add the following lines to servers.lst:

lrnr-ssl-main   main            proxy1.other-life.com   3000    encrypted   Encrypted proxy to the main game server
lrnr-ssl-test   test            proxy1.other-life.com   3001    encrypted   Encrypted proxy to the test server

Using one of these will pop up a warning that the certificate could not be verified because these aren't distributed with the client yet. For now, just ignore the warning and click "Continue".

[feeltheburn]

Update on my end:

• Maps - seem to have gotten some decent testing and feedback. No game breakers, just some minor tweakage. I'll do that over the weekend and send radu the "beta" map fixes for Test around Monday (give or take a couple days).

• Inventory Images - I've made images to handle all "broken/second-hand/damaged/used" weapons/armors, and rare versions of weapons/armor, as well as other stuff that was sharing the same images, fixing a long-time-coming complaint of objects sharing the same image. (Some things like Books, Removal stones, and such will continue to share images as there's too many and they're too limited to bother with.)

• Creatures - I have made a few new looks to differentiate some of the creatures sharing the same look. Assisted by grum in getting them set up properly in the client, Wizzy doing some hidden invading of some to test that don't exist on Test server, and people who have seen screenies of them have so far been positively responding. They'll of course need approval, but the players seem to like the test images so far.

[gvissers]

Learner has indicated that he prefers to support both encrypted an plain connections on the same port. So I have had to adjust both the proxy code and the client code to enable this. Somewhat ironically, I have reverted back to the original design, where the client sends a LETS_ENCRYPT packet to the server on startup, and the server (or proxy, rather), responds with a LETS_ENCRYPT packet of its own. If the server says it does not support encryption, or when no response comes within 5 seconds, the connection is closed.

This is somewhat of a big change somewhat late in the game, so I would really like it if people could test this. As indicated previously, I am currently on vacation, so I may be somewhat slow to respond, but I will follow this issue, and can be reached through the usual means (forum PM ought to work, probably won't be in game though).

[feeltheburn]

I think there's a few issues with black screens and such on the forum that are holding the release for now, but just for my part I'm gonna put a stop to anything I'm doing by mid-September, so if something's not done it just won't go in. I'll pack all the maps and other stuff not in this git up and have it ready for release at that point.

[pjbroad]

@bendoughty, have you built and tried the encrypted client on MacOS?

[bendoughty]

@pjbroad, not yet, unfortunately. I haven't had much spare time recently due to my postgrad dissertation.

The first draft is due next Thursday, so I anticipate I will have some free time to give the new client a spin during the latter half of next week. 😊

[bendoughty]

Everything seems to be working great. I have created a pull request (#152) with an updated Xcode project file.

[feeltheburn]

Running behind on my end, was sick for almost a week among other things.

Sent some final testing maps to radu just now, should be good at this point. I got some tab map work to finish up but it's mostly done, then I'll ball up the pack of data files to use for final 1.9.6 builds and set them up to be downloadable for client builders.

[gvissers]

Not trying to rush anybody, but what's the status? I believe the client code is in good shape, it has actually been compiled and tested on all platforms we support. From what I read here, the maps should be finished as well. Is there still work to be done on the server side?

[feeltheburn]

Actually just tested a final map issue that I sent to Radu yesterday. On my end, I'm final-touching the tab maps (updating the full c1/c2 continent maps that you see in the Tab map view when you click on the upper left corner map image is a bit of a task I forgot I needed to do), then make sure all the updated data files (everything not in this git) are gathered and zipped up for those doing the compiling to download and use.

I shouldn't need more than a week at most, but likely by this weekend.

I'd imagine the eye candy problems ( http://www.eternal-lands.com/forum/index.php?/topic/61631-issues-with-eye-candy-for-example-overly-intense-or-window-darkness-after-cast/ ) are more of a release slowdown issue? Happening in the middle of an invasion, that's the worst time to be getting a bug like that.

[feeltheburn]

My final touches are on track, should definitely be ready with a pack before Wednesday 27th. With one possible exception, may need to get some Rule Changes put in, that may or may not be ready by then, depending on Aislinn and radu hashing them out.

I think we can't release until the eye candy issue is resolved so if it'll take time to work out, so be it.

I'll need to send radu some server stuff to update, but it's only a couple things that would probably take minutes to do just before the server restart for the client update. One file update that I've already done the updates for (and listed what I changed so he can just read and approve, inventory item image changes) and two possible spawn additions for new locations designed specifically for them.

Is there any other outstanding things to be done other than my final pack, the eye candy problem, and last-minute server changes?

[pjbroad]

I've just created a pull request on the data repo with minor command.lst and named_colours.xml additions.

A question for @raduprv is what version number do we want to use for this release. I'm assuming 1.9.6 but could you confirm?

[pjbroad]

@Sir-Odie I've done a pull request on https://github.com/Sir-Odie/EL-Encyc.git for en/commands_help.txt. I presume this is still the official repo.

[pjbroad]

Sorry, another questions. What is the plan to ship a server.lst update and include the certs for SSL connections? Should SSL-Main be the default? I recently committed a client change that enables you to select the default server config. With this in place we could include the certs and extra config lines in servers.lst as its easy to select a new default. Anyone with an existing servers.lst would have to update the file themselves.

[feeltheburn]

My server.lst is a mess of multiple ones for alts and test server and such, so good to know it's not overridden... I should probably test the SSL thing, hmm. I'll compile an update later today.

I'll be grabbing Odie's encyc git at the last minute for the final data file package, so yeah, adding stuff there will be in the final pack.

I've mentioned 1.9.6 to radu several times with him saying nothing about it. I don't think he's caring about the numbering, so I'd personally say go with 1.9.6. (Or just assume it unless he comes in screaming for a different number, hehe.)

My status: I'm still on track for having a data pack ready by Wednesday. Doing some last-minute visual scans of the new tab maps to look for errors now but they seem fine so far. What's left is mostly just grabbing everything and putting it together... data files from the private gitlab git, maps, encyc from Odie's git. And hopefully Aislinn and Radu will have worked out rule updates by then, it's the last outstanding part of the data.

Is a full list of changes to the client since 1.9.5 still kept that players can mostly understand? Radu will want that when the update happens. I have a list for maps and data files.

[raduprv]

Hmm, it seems that just replying to the e-mail doen't work, I have to post the comments on github. Yes, 1.9.6 is fine. I was also asking about the Android APK if it's ready, I'll have to sign it with my own key. As for the server list, let's leave it as it is for now, without the SSL stuff, since it wasn't extensively tested. People who want to test it can add things manually there.

[pjbroad]

Looks like we found and fixed the eye candy issue so we're ready for the final stages.

The Android client source is up to date with the main client. @raduprv if you're not doing to build it yourself, could you resign or unpack and resign a build I do for android? I also have have build instructions and scripts in this repo that you might want to try. The Android client now supports CUSTOM_UPDATE and CUSTOM_LOOK which you may want to turn off for the play store.

[raduprv]

I guess I can give you the key thing, it's mainly used for the EL binary anyway (and some other app I have on the Playstore that I don't care very much about).

[feeltheburn]

I'm almost done. Bluap, check forum PM.

I'll have the server change list sent to radu later today (not much to do), and updated data file pack available for download as well.

Er... that android build needs these new data files, too. ;)

It's a combo of the private el data git which is getting a last minute update, the private map file git, and Odie's encyclopedia git. Git 'er done...

[pjbroad]

I guess I can give you the key thing, it's mainly used for the EL binary anyway (and some other app I have on the Playstore that I don't care very much about).

That's fine. Not sure of the best way to send it to me but you have my email.

On the SSL stuff. It's been 100% stable for me and I know Aislinn has been using it too. Can we at least included the server certificates in the data pack? My preference would be to also add additional lines to the server.lst file too, but not necessarily enable it as the default.

[raduprv]

Yes, we can introduce the certificates, but not the server list for now. Those who wish to change it can do it by hand, but I don't want it to be available for everyone just yet. Btw, the Android APK needs to be under 100mb in size, or else we can't host it on Google Play. So some non essential tab maps will have to be removed or downsized.

[feeltheburn]

Right. Based on what bluap's current build size is using standard tab maps, the special reduced-size tab map pack I have put together just for Android will put it well below that without removing any*. So that's covered.

*Any that regular players can access, that is. I removed unused-by-players maps like testermap and neo.

[feeltheburn]

bluap - As soon as I get the certificates in the data git (or otherwise if they shouldn't be there for some reason) I'll post the data pack and the android map pack.

[pjbroad]

bluap - As soon as I get the certificates in the data git (or otherwise if they shouldn't be there for some reason) I'll post the data pack and the android map pack.

Thanks, merge request done.

[feeltheburn]

DATA PACK

https://el-db.com/tempstuff/196release/el_data-196.zip

ANDROID MAP PACK

https://el-db.com/tempstuff/196release/maps-196-FOR-ANDROID-ONLY.zip

IMPORTANT: For Android, it includes everything needed so completely delete the /maps/ folder in the data pack and replace it with this, don't just overwrite. Some unusable-by-player maps don't appear in the android pack that do in the data pack, overwriting would keep them and their larger tab maps.

[gamil-zirak]

DATA PACK

https://el-db.com/tempstuff/196release/el_data-196.zip

The cyclops_2.xml file is missing causing an error preventing the game from loading. The other new actor definitions are present though.

[feeltheburn]

FIXED. There is no cyclops_2 files, that was something I was testing but didn't finish.

You can redownload, or open el_data/actor_defs/actor_defs.xml, find the 2 lines that reference cyclops_2, and delete them.

[gamil-zirak]

Thanks, that fixed it.

[pjbroad]

It's minor I know but I was just comparing the data zip with the versions in the client dev directory and noticed in item_info.txt Turquoise is missing the second "u".

It's probably time to bump the version numbers for the client and build some test packages. @raduprv do you want to up the server version numbers? Currently version_first_digit=10 and version_second_digit=29 (or 28 if compiled without SSL).

[raduprv]

version_second_digit=29 sounds good.

[pjbroad]

version_second_digit=29 sounds good.

Its already 29 if compiled with USE_SSL (the default), but 28 if compiled without USE_SSL. So leave it at that or bump both to 29?

[raduprv]

28 is the current one, so they both must be upgraded to 29

[feeltheburn]

The item_info is script-built directly from the items.def file that's on the server. Which has always been without the second U according to my harv stats. I didn't wanna touch that because it'll mess with counters.

(Spelling-wise in the server items list, I only fixed 2: "raccoon hat" got its missing C because it's not something that'll be high on anyone's counters, if it's on them at all, and "Wilhelm Hood Removal" stone because it was impossible to search for with an L missing in the name, and there's no counter issues.)

[pjbroad]

The item_info is script-built directly from the items.def file that's on the server. Which has always been without the second U according to my harv stats. I didn't wanna touch that because it'll mess with counters.

I only mention it as the previous item_info.txt file shipped with the client spelt it correctly. The description in that file is not used by the counters as they scrape the name from the server messages "you made a -thing-". It's not a big deal.

[raduprv]

Bluap, which e-mail address shall I use to send you the key file? I have 3 addresses for you, not sure which one is your primary one.

[pjbroad]

Bluap, which e-mail address shall I use to send you the key file? I have 3 addresses for you, not sure which one is your primary one.

I've sent you an email.

[feeltheburn]

@pjbroad bluap, have you tested the Android build size with the special tab maps for it?

[pjbroad]

@pjbroad bluap, have you tested the Android build size with the special tab maps for it?

Just tried it, apk size is 87M so its fine.

[feeltheburn]

Okay, coolness. They should be okay but holler if you test it and the tab maps aren't showing well. We can try some larger (pixelwise) jpg versions that might be better.

...

I've posted randomly multiple times in @@6 to try and get people to test the eye candy issue over several days, don't think we're gonna get any more to check it. Those of us that have tested it are all inclined to think it's fixed.

The client otherwise is very well tested with people actually using bluap's builds or like me self-compiling the git and using it in game play daily.

The updates I sent radu for server was a quite short list that shouldn't take long to do at the last minute.

Are we waiting on anything else at this point? Can we finally do this? ;)