incorporate upstream changes into polkadexTEE-worker

Integritee-worker is being continuously upgraded. Some changes are unrelevant for the polkadexTEE-worker, but some are certainly helpfull improvements.

In this issue we should discuss how we should handle upgrading to the upstream branch. The following questions should be answered: 1) Do we even want to follow suit of the integritee-worker?

2) If yes: a) Should we rather cherry pick commits that are useful for us, and leave out others? (with the danger of more merge conflicts later on) or b) should we just sometimes update & merge to the most recent upstream master?

The problems I see are the following: Option a) : we will have to check every single commit & review the cherry-pick as a PR again in the polkadexTEE worker. We would also need to decide who should decide on which commits should be integrated and which not. Option b): That will create a huge PR, in case we wait for a little bit longer, increasing the chance of introducing some errors, especially if we don't have integration test that fully cover all use cases of the worker.

As decided in the Roadmap call (18.10.2021) I will summarize what changes have been done upstream and are useful for polkdex. We can then decide together if we want to integrate them or not based on this summary.

I'm recommending to integrate the following commits (starting from 23th of august, PR #288 already includes older updates)

[x] 23.8: https://github.com/integritee-network/worker/commit/ebe693f4d05948c2989fae73f55a6e7f351f5c7f : small commit, abstracts Aes and Ed25519 into separate crate , including a more specific trait bound
- [x] 27.8: https://github.com/integritee-network/worker/commit/3b517d6ddcf006def9064efebad6ccdaffa39479 rename chain_relay to more expressive light_client and exstract to separate crate, making it possible to test with cargo test
[x] 1.9: https://github.com/integritee-network/worker/commit/d4495b8eba208161e646c1445f3a1a4ee0f2393a makes the prefuning of the enclave more generic. Unsure if needed, but it makes the enclave funding easier to adapt
[x] 2.9: https://github.com/integritee-network/worker/commit/39f235d368c224dcff0f3b47f0db5b49a8eaa061 rebranding & renaming. I'd incooperate the library renaming (worker -> service, enclave -> enclave-runtime, making it more substrate node aligned), the rebrading part I'd exclude, Rather: Should we do some polkadex rebranding?
[x] 7.9: Enclave Ocall api clean up : https://github.com/integritee-network/worker/commit/35b303426e289d3f7d10b3ac382c10df4763e0df (better description here: https://github.com/integritee-network/worker/pull/382)
[x] Partially include some of the renamings of: https://github.com/integritee-network/worker/commit/8200b58936335c2e40c7ca4b582227002b9153b3 (itp_core -> itp_types, primitives -> core-primitives, client -> cli)
[ ] 10.9 Small simlink fix in the cli: https://github.com/integritee-network/worker/commit/ccd2be65002323a9961d719d42232b7bcca5c187
[x] 10.9 https://github.com/integritee-network/worker/commit/226167ea3cc1778b27cf03d3b74f4768df11ddd8 move OpaqueCalls to itp_types
[x] 14.9: Run RPC websocket entirely inside enclave: Very important one but one that will require some manual merge confict solving as well as benchmark testing: https://github.com/integritee-network/worker/commit/0af6918671fc130cfe3780e0a3c783ad59cf72d9
[x] 16.9 https://github.com/integritee-network/worker/commit/a9cc1c3ee5341b618c5c828f5569a72c2e01ede5 no extra functionality, but it brings us one step closer to an optional sidechain which will let us completely remove this feature without any extra surgery.
[x] 21.9: https://github.com/integritee-network/worker/commit/03c3e593be5fdc1b4a10327ff13fb4b56c1f70d6 abstraction of rsa3072, making it easier to exchange in case it's needed
[ ] 21.9: Addition of REST-client (for non-repudiation use cases): https://github.com/integritee-network/worker/commit/03c3e593be5fdc1b4a10327ff13fb4b56c1f70d6 - do we need this?
[x] 23.9: adds pin sgx into the make file: https://github.com/integritee-network/worker/commit/1181835364c14eee771521ce268b49b3b51bbbe7 (copy past is probably faster than a cherry pick)
[x] 23.9: make file fix: https://github.com/integritee-network/worker/commit/564c137693a9bfec3e395255223c33cce85d8580
[x] 24.9: enclave test clean up: https://github.com/integritee-network/worker/commit/c581f180c8dcac5dafc1337c2d05a806cc6eea2e
[x] 28.9: refactor top pool author to use a facade : one step closer to top-pool removal: https://github.com/integritee-network/worker/commit/502a4e2dddc947ce6cdbd247d7103387bb1872b8 & https://github.com/integritee-network/worker/commit/717b066c306750ac8d2328373b8deed40fa05d0e
[x] 29.9: fix reexport in itp-types: https://github.com/integritee-network/worker/commit/88d91571b5fd982be32be7397ad19d28302c30d4
[x] 01.10: https://github.com/integritee-network/worker/commit/c60f460bf3ce2d019cb78e675ce19b0f6df1abc8 includes alot of sidechain stuff which we will not need, but some light-client changes were made in there, which we should incooperate.
[x] 10.10: state abstraction (facade) introduction: https://github.com/integritee-network/worker/commit/68ffa150e806040e1e284276d726fed1054d96c7 one step closer to complete removal
[x] 10.10: remove some println stuff: https://github.com/integritee-network/worker/commit/65c555babcfdb723d652a841d6c8e85f051a396e
[ ] 10.10: demo fix: https://github.com/integritee-network/worker/commit/e4c454ea3833e2213ae917b90329b186fc0bd3e6 introduction of timeouts to ci test scripts: need to look to see if we need something similar for our CI? & https://github.com/integritee-network/worker/commit/f16c4a91ac2548dd903703cade31918363e840e4
[x] 11.10: https://github.com/integritee-network/worker/commit/987187691b3ad912792a9fdff53b7d18d7065e9e add some more abstraction to stf state
[x] 11.10: CI fix, wont hurt to include: https://github.com/integritee-network/worker/commit/876f1fd1b1f5af5e72343fa810f224f30c2d0504
[x] 12.10: separates parentchain sync & block execution: https://github.com/integritee-network/worker/commit/3d3e93c42d939f21bcde3d0a22e6b4b768fd65ac (very interesting for us, as it will help to remove sidechain)
[x] 13.10: enable production mode build of enclave : https://github.com/integritee-network/worker/commit/338015fff43392af61a64c32d1bbdce5e0de1851 & hot fix of it: https://github.com/integritee-network/worker/commit/29bd6d3e516cd12f0ac763fad10106b003d38884

General comment: All these extractions & abstractions to extra crates actually help to make the whole codebase better unit testable. A separate crate can be easily tested with cargo test and included in enclave with no-std import when needed in there. (cargo test not available inside encalve, making it hard to unit test)

What we do not need to include:

27.8: https://github.com/integritee-network/worker/commit/05f1fef5e5d86c568a50792563c2ee6d9c5b8c91 sidechain specifc, not used by polkadex
4.9: https://github.com/integritee-network/worker/commit/57adccc66b2b37737d5d4af3bd6d7f48c9fcca99 sidechain
6.9: https://github.com/integritee-network/worker/commit/47dca23076e8039dad42c0739220c9aaf92da6a7 phyton script, not used by polkaday
9.9: https://github.com/integritee-network/worker/commit/d47ee96a7f7aed628ef08bbc26017eef43806f10 unimportant GHA bump
10.9: https://github.com/integritee-network/worker/commit/38d8d7d7c605448ff54932fa043c03cd3c4c0080 sidechain specific
16.9 : https://github.com/integritee-network/worker/commit/ebf0a6fc760f0fd1a015e81b0a8fe6434f7e1555 fix for block_production which we are not using
22.9: https://github.com/integritee-network/worker/commit/692ff071c086d1e50cef0a3af3cca738389dba05 imporvement of stf state, which we are not using
05.10: https://github.com/integritee-network/worker/commit/d1476600e1ba4e451ec61dec62e2f3b57eedf2cd stf clean up, which we are not using anyway
06.10: https://github.com/integritee-network/worker/commit/ef71dc60f21330526e38d39450b8a6903a0ac745 sidechain consensus, not used by polkadex
10.10: https://github.com/integritee-network/worker/commit/31fca802eae898985b1e329f99a66a2b2cad8bcf sidechain
10.10: https://github.com/integritee-network/worker/commit/1a65938ebad83ff0831cb07e341bd3583c9b8e9f hot fix of CI which will be reverted later on
11.10: https://github.com/integritee-network/worker/commit/d796e46cc67a9baf7edb73f89451f79e1e1ebb46 addition of author filter, not used by polkadex
12.10: removal of above mentioned hotfix: https://github.com/integritee-network/worker/commit/d4d563fe4f7401924bc1057296118769531ef2ba
13.10 Stf clean up: https://github.com/integritee-network/worker/commit/b8c9b1ba104e943b1e1935f4c8b52d2ff05977de
13.10: load intialised state: https://github.com/integritee-network/worker/commit/139a5845afd2119959f2beeb9f700e22c0f3a417 clean up some code but code we don't need
13.10: https://github.com/integritee-network/worker/commit/f15ec4cb73461183e375a4ce88a409ddeef78fc0 already integrated in polkadex
19.10: https://github.com/integritee-network/worker/commit/48459c167eb2bdaafe94231c7bd10662e129d452 stf trusted getter

@simonsso is the above list somewhat sensible to you? And if yes, can you agree to the separation I made?

TODO include:

[x] 23.8: https://github.com/integritee-network/worker/commit/ebe693f4d05948c2989fae73f55a6e7f351f5c7f : small commit, abstracts Aes and Ed25519 into separate crate , including a more specific trait bound -> PR #317
- [ ] 27.8: https://github.com/integritee-network/worker/commit/3b517d6ddcf006def9064efebad6ccdaffa39479 rename chain_relay to more expressive light_client and exstract to separate crate, making it possible to test with cargo test
[ ] 1.9: https://github.com/integritee-network/worker/commit/d4495b8eba208161e646c1445f3a1a4ee0f2393a makes the prefuning of the enclave more generic. Unsure if needed, but it makes the enclave funding easier to adapt
[ ] 2.9: https://github.com/integritee-network/worker/commit/39f235d368c224dcff0f3b47f0db5b49a8eaa061 rebranding & renaming. I'd incooperate the library renaming (worker -> service, enclave -> enclave-runtime, making it more substrate node aligned), the rebrading part I'd exclude, Rather: Should we do some polkadex rebranding?
[ ] 7.9: Enclave Ocall api clean up : https://github.com/integritee-network/worker/commit/35b303426e289d3f7d10b3ac382c10df4763e0df (better description here: https://github.com/integritee-network/worker/pull/382)
[ ] Partially include some of the renamings of: https://github.com/integritee-network/worker/commit/8200b58936335c2e40c7ca4b582227002b9153b3 (itp_core -> itp_types, primitives -> core-primitives, client -> cli)
[ ] 10.9 https://github.com/integritee-network/worker/commit/226167ea3cc1778b27cf03d3b74f4768df11ddd8 move OpaqueCalls to itp_types
[ ] 14.9: Run RPC websocket entirely inside enclave: Very important one but one that will require some manual merge confict solving as well as benchmark testing: https://github.com/integritee-network/worker/commit/0af6918671fc130cfe3780e0a3c783ad59cf72d9
[ ] 16.9 https://github.com/integritee-network/worker/commit/a9cc1c3ee5341b618c5c828f5569a72c2e01ede5 no extra functionality, but it brings us one step closer to an optional sidechain which will let us completely remove this feature without any extra surgery.
[ ] 21.9: https://github.com/integritee-network/worker/commit/03c3e593be5fdc1b4a10327ff13fb4b56c1f70d6 abstraction of rsa3072, making it easier to exchange in case it's needed
[ ] 23.9: adds pin sgx into the make file: https://github.com/integritee-network/worker/commit/1181835364c14eee771521ce268b49b3b51bbbe7 (copy past is probably faster than a cherry pick)
[ ] 23.9: make file fix: https://github.com/integritee-network/worker/commit/564c137693a9bfec3e395255223c33cce85d8580
[ ] 24.9: enclave test clean up: https://github.com/integritee-network/worker/commit/c581f180c8dcac5dafc1337c2d05a806cc6eea2e
[ ] 28.9: refactor top pool author to use a facade : one step closer to top-pool removal: https://github.com/integritee-network/worker/commit/502a4e2dddc947ce6cdbd247d7103387bb1872b8 & https://github.com/integritee-network/worker/commit/717b066c306750ac8d2328373b8deed40fa05d0e
[ ] 29.9: fix reexport in itp-types: https://github.com/integritee-network/worker/commit/88d91571b5fd982be32be7397ad19d28302c30d4
[ ] 01.10: https://github.com/integritee-network/worker/commit/c60f460bf3ce2d019cb78e675ce19b0f6df1abc8 includes alot of sidechain stuff which we will not need, but some light-client changes were made in there, which we should incooperate.
[ ] 10.10: state abstraction (facade) introduction: https://github.com/integritee-network/worker/commit/68ffa150e806040e1e284276d726fed1054d96c7 one step closer to complete removal
[ ] 10.10: remove some println stuff: https://github.com/integritee-network/worker/commit/65c555babcfdb723d652a841d6c8e85f051a396e
[ ] 11.10: https://github.com/integritee-network/worker/commit/987187691b3ad912792a9fdff53b7d18d7065e9e add some more abstraction to stf state
[ ] 11.10: CI fix, wont hurt to include: https://github.com/integritee-network/worker/commit/876f1fd1b1f5af5e72343fa810f224f30c2d0504
[ ] 12.10: separates parentchain sync & block execution: https://github.com/integritee-network/worker/commit/3d3e93c42d939f21bcde3d0a22e6b4b768fd65ac (very interesting for us, as it will help to remove sidechain)
[ ] 13.10: enable production mode build of enclave : https://github.com/integritee-network/worker/commit/338015fff43392af61a64c32d1bbdce5e0de1851 & hot fix of it: https://github.com/integritee-network/worker/commit/29bd6d3e516cd12f0ac763fad10106b003d38884

Polkadex-Substrate / polkadexTEE-worker

incorporate upstream changes into polkadexTEE-worker #304