IPv8 distributed apps / plugins

[synctext]

Goal: Distributed apps with efficient execution model and event-processing.

Possible operation scenario: you download some blockchain "genesis code" to connect to the blockchain. You now can download the basic data and complete the bootstrap. This basic data would somehow be trustworthy and executable. Trust and abuse prevention is the key challenge.

this work fits into our long line of work around this theme.

Plugins and apps have been around for decades. In 1999 I've created one within Winamp. Numerous app ecosystems emerged, in 2008 a vulnerability was published in Facebook Apps. Dealing with vulnerabilities, spam, and malware has proven to be a hard problem. Tezos aims to have a self-describing blockchain, but they did not yet address any of the fundamental security issues.

In 2008 the Tribler research team reached a key milestone: an app store and execution platform without any server. See our research report on this full self organising system, this work has a distant relation to the Ethereum smart contracts which launched years later. Real 2008 widget code without any servers, no app store and no app platform servers. In 2015 we created self-compiling Android apps ("Autonomous smartphone apps: self-compilation, mutation, and viral spreading").

Widget platform	estimated number of widgets	estimated average size per widget (KB)	estimated total size (MB)
Eclipse Plugins	1,200	900	1,054
Facebook Apps	40,000	–	–
Firefox extensions	3,100	500	1,500
Google Desktop Gadgets	1,250	80	100
Windows Vista Gadgets	5,800	140	780
Yahoo Widgets 5,000	600	2,930

Estimated repository size of various widget platforms in 2008

The 2008 widget source code was simply gossiped around between clients. Each client collected raw .py files and displayed a local app store interface like:

Our 2008 architecture for serverless widgets:

full 2009 thesis

[synctext]

[Copied from ticket #3670, moving activity here; oldest ticket] Master thesis idea:

• permissionless execution of Python code
  • No permission needed
  • Removes all dependency from servers
  • architecture without any single-point-of-failure
  • Open innovation and module growth
  • source code distribution and forking by design
• how to build trust for dAPP developers
  • star ratings for dAPP authors
  • code reviews and endorsements
  • prevent fake vote flooding (e.g. Sybil attack)
  • re-use Trustchain
• IPv8 execution engine
• dAPP discovery and update service
  • display all discovered dAPPs
  • View and click-to-execute untrusted code
• package manager
  • Python code is published as a swarm
  • Source code inside a bittorrent swarm
  • info_hash as a single requirement to identify, download, install, and run code.
• runtime environment
  • event-based intra-package communication bus (local-only)
  • dAPP-to-dAPP overlay, each running app joins the IPv8 overlay with the same app
  • dAPP version updates are gossiped around on this overlay
  • dAPPs: Loading Python modules/plug-ins at runtime

Possible sprints:

• seed swarm with Hello World; import and run
• add IPv8 and execution engine V1
• gossip random list of hashes of discovered code.
• implement client-server protocol of popular Android apps and possible re-use GCM push tokens

[synctext]

Security is a key concern In the first sprints I would recommend focusing on the core features first. Explore security after that. There are sandboxing features like RestrictedPython and chrootbuilder, etc. A "Sandboxed Python" would let you permit or forbid modules, limit execution slices, permit or deny network traffic, constrain filesystem access to a particular directory (floated as "/"), and so on.

Fun part: IPv8 execution engine is itself obviously an IPv8 distributed app. Can upgrade itself.

[synctext]

security threat observed in the wild: plugins(docker images) which steal cpu cycles for mining: https://arstechnica.com/information-technology/2018/06/backdoored-images-downloaded-5-million-times-finally-removed-from-docker-hub/

[synctext]

Essentially, the public Internet is now a platform of apps on mobile devices

[synctext]

Towards first running code prototype:

• cmdline simplicity
• Libtorrent swarm, filled with execute.py source code file
• download hard-coded swarm
• unpack this swarm
• magical import and execute.
• possibly different executing approaches with different modules/packages/sandboxes.
• Hello World!\n

Goal: self-sustainable ecosystem. Dream outcome: Runs flawlessly and expands after thesis is completed :-)

[synctext]

Status update: experimented with IPv8, learning about related work.

• part-time thesis for now, double head TA.
• Dynamic loading: http://www.diveintopython.net/functional_programming/dynamic_import.html
• IPv8 as service or as a library?
  • 'the network is the code' :-)
  • each dApp has its own overlay (intra-dApp-overlay)
  • dAPP discovery and update service (inter-dApp-overlay)
• Advice: first running code with Libtorrent, dynamic import, then later expand with IPv8 magic
• note: the mechanisms are easy to fix and repair once you find out they are inefficient. You will not get it right the first time. Policies are much harder and difficult to understand. Their is no textbook on how to do this successfully. Key challenge is the governance, audit, and management layer, for instance, CPAN stats: "174,504 Perl modules in 38,832 distributions, written by 13,592 authors, mirrored on 263 servers".
• Early (social) design decision: anonymous code uploading not allowed (security risk)
• @mitchellolsthoorn : the balance between experimenting, exploration, and engineering.
• far future example Facebook-alternative dApp
• Next Meeting: code demo from open Github repo

[synctext]

Let's just go on another dreaming excursion...:

IPv8 dApps are the fundamental building blocks for the re-decentralisation of The Internet. It represents an evolution from monolithic code, REST-based micro-services, towards trustworthy global computing. For over 20 years we have dreamed of composable software. We expand upon the idea of smart contract, by making them scalable and smarter. We never had software repositories which are server-free and maintenance-free. Now we finally have build the first autonomous isolated software pieces which understand the foundations of trust, can made decisions and learn, manage their own money, trade on marketplaces, can self-replicate, can do self-compilation, self-organise online elections, and have long-term reputational memory. Basic business dApps can be build for electronic signatures, legal document handling, product catalog publication, invoice processing, ordering, inventory finance, currency conversion, bank account management, micro-loans, and other basic banking functions.

Possible sprints to brainstorm about after Libtorrent basics are operational (our relentless incremental improvement methodology :-)

• dApp runtime support and communication paradigm
• dApp event communication
• Prototype for a week with (I'm assuming these are superior to old-skool REST):
  • https://pypi.org/project/amqp/
  • https://github.com/pika/pika
  • https://pika.readthedocs.io/en/0.9.13/modules/adapters/twisted.html
  • Read about Kafka, ActiveMQ, and Kestrel?

Even more ambitious is the offline-first approach, like Bramble, but they lack structural funding. Bramble builds upon Tor and handles all of the core functionality of "sending blocks of data back and forth, managing contacts, keeping channels between users secure and metadata-free, synchronizing state between a user's devices, and handling dependencies between pieces of data or expiring them when they get too old".

[mitchellolsthoorn]

Got the first prototype to work. https://github.com/mitchellolsthoorn/ipv8-dapps-loader/

A torrent file is created from the 'execute.py' file which contains a hello world code example. This torrent is seeded on the local network by the 'seeder.py'. The loader downloads this torrent and loads and runs the first dapp of this system

[synctext]

Solid progress! Relentless incremental improvement, next step: trust, community building and code-reviews?

Brainstorm outcome: we use Github rating of software developers as a metric for quality of a code review. Just for initial bootstrapping. In later iterations we could hide the identity of the reviewers and have randomly picked witnesses validate the reputation of a software developer. Requirements for core functionality:

• a single code review is published in local Trustchain (sign hash of dApp with private key)
• app voting and discovery overlay (MvP: gossip 5 votes for dApps you reviewed)
• Can we safely import a limited-length unicode string from a malicious dApp (a.k.a. "dApp name")?

[synctext]

Brainstorming about the next upcoming sprint after skeleton voting is operational...

2805 ticket holder @alexander-stannat has now been volunteered to focus on creating a dApp of trust! Goal is to provide visualization and calculation of trust. It would require an API for real-time data feed of Trustchain, GUI events, and user selection of nodes. Upon selecting a node it would display the public key of that node and current trust estimate. Please work together when ready for that.

[synctext]

Related work, "Crev: dependency vetting with a web of trust" (Alpha stage), https://news.ycombinator.com/item?id=18824923

[mitchellolsthoorn]

Progress update: Voting logic is implemented.

At the moment, nodes vote (+1) on dapps when they receive newly spreaded dapps. The dapps are gossiped to all nodes once. The gossiping happens through voting, where popular or good dapps are gossiped more than others.

I ran a successful experiment where all selected dapps where discovered by the nodes set-up.

Possible plans for next sprint is:

• storing all discovered dapps in a DB
• make the discovered dapps visible
• make the user able to vote on dapps, instead of doing it automatically
• running an experiment with all components across multiple computers
• make an interface for dapps, instead of the current file with code

[synctext]

Meeting minutes:

• execution of .python inside a Bittorrent swarm
• Blockchain-based voting inside our IPv8 overlay
• uses Trustchain in IPv8: _trustchain_peer = Peer(ECCrypto().generate_key(u"curve25519"))_

  ipv8 = IPv8(configuration)
  trustchain_peer = Peer(ECCrypto().generate_key(u"curve25519"))
  trustchain_community = TrustChainTestnetCommunity(trustchain_peer, ipv8.endpoint, ipv8.network, working_directory='./'+str(i))
  ipv8.overlays.append(trustchain_community)
  ipv8.strategies.append((EdgeWalk(trustchain_community), 10))
  dapp_community = DAppCommunity(trustchain_peer, ipv8.endpoint, ipv8.network, trustchain=trustchain_community)

• cmdline-only sprint
• Is March Public Beta launch still an option? (stand-alone)
• Launch as an Android developers power-shell with code dissemination and trust?
  • runtime-import("Bittorrent-SHA1");
  • discovered-DApps();
  • normal shell commands
  • power-prompt toolkit seems one of many offerings Android?
• GUI solution for interaction?
  • runtime addition of GUI elements, screens, requester prompts and popups.
  • Usable with QT5, Kivy, Java or need for a domain-specific language: DApp-GUI?
  • Showstopper, for instance, need to compile QT5 for iOS when distributed in objective C.
  • Google to the rescue with Flutter
• Future sprints: Dart Hello World from inside swarm
• Downloads all DApps by default upto 10MByte maximum and seed
• Solve availability problem, but only vote if source code is reviewed and approved
• Future sprints: only download and seed if sufficient votes are cast (DDoS)
• Future sprints: multiple trust functions

ImportError and/or path error:

>python loader/runner.py 
Traceback (most recent call last):
  File "loader/runner.py", line 5, in <module>
    from community.dapp.community import DAppCommunity
  File "/home/pouwelse/GITHUB/ipv8-dapps-loader/loader/community/dapp/community.py", line 1, in <module>
    from block import DAppBlock
  File "/home/pouwelse/GITHUB/ipv8-dapps-loader/loader/community/dapp/block.py", line 1, in <module>
    from pyipv8.ipv8.attestation.trustchain.block import TrustChainBlock
ImportError: No module named pyipv8.ipv8.attestation.trustchain.block

[synctext]

Meeting minutes & progress:

• twistified, big refactor
• cmdline operational
• permanent database (no longer simple in-memory)
• next sprint: a simple Hello World and another Trustchain crawler service!
• cross-platform with Android support, possibly Android Webview {default GUI, with IPv8 services in background; dynamic GUI update)

Works!

2019-01-22T12:28:01+0100 [-] 
            .___  _____                         
          __| _/ /  _  \  ______  ______  ______
         / __ | /  /_\  \ \____ \ \____ \/  ___/
        / /_/  /    |    \   |_> >   |_> >___ \ 
        \____  \____|__  /    __/ |   __/____  >
             \/        \/ |__|    |__|       \/ 

2019-01-22T12:28:01+0100 [-] version 0.1
2019-01-22T12:28:01+0100 [-] [0] Show dApps
2019-01-22T12:28:01+0100 [-] [1] Create dApp
2019-01-22T12:28:01+0100 [-] [2] Exit
0
2019-01-22T12:28:03+0100 [-] Unhandled Error
    Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 103, in callWithLogger
        return callWithContext({"system": lp}, func, *args, **kw)
      File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 86, in callWithContext
        return context.call({ILogContext: newCtx}, func, *args, **kw)
      File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 122, in callWithContext
        return self.currentContext().callWithContext(ctx, func, *args, **kw)
      File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 85, in callWithContext
        return func(*args,**kw)
    --- <exception caught here> ---
      File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
        why = selectable.doRead()
      File "/usr/lib/python2.7/dist-packages/twisted/internet/process.py", line 291, in doRead
        return fdesc.readFromFD(self.fd, self.dataReceived)
      File "/usr/lib/python2.7/dist-packages/twisted/internet/fdesc.py", line 94, in readFromFD
        callback(output)
      File "/usr/lib/python2.7/dist-packages/twisted/internet/process.py", line 295, in dataReceived
        self.proc.childDataReceived(self.name, data)
      File "/usr/lib/python2.7/dist-packages/twisted/internet/_posixstdio.py", line 77, in childDataReceived
        self.protocol.dataReceived(data)
      File "/usr/lib/python2.7/dist-packages/twisted/protocols/basic.py", line 571, in dataReceived
        why = self.lineReceived(line)
      File "/home/pouwelse/GITHUB/ipv8-dapps-loader/twisted/plugins/cli_plugin.py", line 84, in lineReceived
        self.menu_items[int(line)].values()[0]()
      File "/home/pouwelse/GITHUB/ipv8-dapps-loader/twisted/plugins/cli_plugin.py", line 107, in show_dapps
        msg(self._colorize("info_hash: " + dapp['info_hash'] + " name: " + dapp['name'], 'green'))
    exceptions.TypeError: cannot concatenate 'str' and 'buffer' objects

[mitchellolsthoorn]

Sprint Progress:

Fixed issues discovered during the current and previous sprint

The issues with the CLI interface and the menu were fixed. There were some conversion errors when printing out values from the database, which caused the errors seen on the previous sprint update. The program was also receiving a lot of packet malformatted errors on network packets. This turned out to be caused by an increased block size type that wasn't fixed in the Test TrustChain Network. A PR with an update was created and merged in IPv8 and uploaded to pip.

Database / Verification checks

The database was updated to store more structured data, so the program doesn't have to query the unstructured trustchain every time. This change also allowed for better checking of software guards like if the current dApp is known to the system before executing an action on it, and preventing double votes.

By making this change, there could grow inconsistencies between the trustchain and the database. This could happen through manual tampering of the database or trustchain, or through glitches in the blocks received flow. To prevent this, verification checks are executed 5 seconds after start-up, that detect these inconsistencies and fix them.

Automatic dApp crawling

The previous distribution mechanism of dApp loader only distributed dApps to connected peers when:

• A node created a new dApp
• A node voted on an existing dApp

This strategy could lead to a dead network with passive users and a slow onset time for dApps. To help solve this problem a dApp crawler was created and added, that searches connected peers for unknown dApps, to decrease this onset time. Since the crawler only crawls the first level connected peers, it doesn't impact the scalability significantly.

The crawling task is executed asynchronously 10 seconds after start-up and every hour afterward.

Trust - Double vote detection

To prevent malicious nodes from infecting the network with fake votes to promote certain dApps, a double vote detection check has been added to detect these. Upon detection, a message is shown to the user of the system to indicate this. This double vote detection happens on start-up but could be adjusted to run more often.

Libtorrent integration

In the previous sprint the program consisted out of multiple parts:

• dApp community
• loader
• seeder
• runner

These parts have now all been integrated into the dApp community. This integration led to some problems with the (dynamic) package import, but from preliminary testing, it should all work now. Context options were added to the menu to reflect these:

Main Menu

• create test dapp (creates a fake test dapp package for testing)
• create dapp (creates a dapp from a dapp package with the package name provided by the user, this package should be located in the payloads directory in the state folder e.g. state_dir/payloads/package_name)

  Context Menu (only after discovery)

• download dapp (downloads the dapp through libtorrent)
• run dapp (runs the dapp - first has to be downloaded)

Execution engine implementation

In the development of the execution engine, several problems were encountered:

• Overlays have a lot of common services that could conflict e.g. Trustchain
• These common services degrade the security and isolation of the running dApps e.g. by accessing each others components directly
• Overlays have different types of hierarchical levels e.g. service vs overlay component
• Some overlays use adapted common services e.g. Tracker
• dependencies are difficult to handle

hierarchical types

• Executable This was the basic test approach, but could be useful for certain dapps

• IPv8 overlay This is the original idea of adding overlays to the same IPv8 instance that dApp loader is using. This, however, causes problems with common services like TrustChain that need to be adapted for certain dapps. Also, port numbers would be more challenging to manage if extra services are started e.g. REST manager. This type is however needed/preferred when running multiple modularised application modules.

• Twisted service Twisted services could provide an alternative or secondary approach, where each service would run their own IPv8 instance and thus run on its own port. This would eliminate the common services problem and increase the security, but also the overhead.

Experiment - Executable: Distributing and running of test dApp

Experiment - Overlay: Distributing and running of IPv8 Crawler

Experiment - Service: Distributing and running of IPv8 Tracker

[synctext]

Success! :smile: Operational Trustchain crawler as a dApp. Meeting minutes, focus on either market dApp, trust dApp or implement dynamic GUI. Best focus on the toughest problem, with roadmap consequences: GUI stuff.

Sprint goal: in 2 weeks make prototype of dynamic GUI. prior notes "cross-platform with Android support, possibly Android Webview {default GUI, with IPv8 services in background; dynamic GUI update)". Prior work of QR code scanning and reading in Android. Great basic dApp dependency.. Test for Android: Trustchain crawler GUI.

Create demo .apk (ipv8-service and IPv8 app)

Future sprints: market dApp, Bitcoin wallet, QR-codes + IBANs. (ring-fenced local and global REST api)

2019-02-04T14:26:12+0100 [-] version 0.1
2019-02-04 14:26:12,698:DEBUG:dApp-community: Getting all dApps from catalog
2019-02-04 14:26:12,698:DEBUG:persistence: Getting all dApps from catalog
2019-02-04T14:26:12+0100 [-] 0 dApps found:
2019-02-04T14:26:12+0100 [-] [-1] Return to previous menu
0
2019-02-04T14:26:15+0100 [-] Unhandled Error
    Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 103, in callWithLogger
        return callWithContext({"system": lp}, func, *args, **kw)
      File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 86, in callWithContext
        return context.call({ILogContext: newCtx}, func, *args, **kw)
      File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 122, in callWithContext
        return self.currentContext().callWithContext(ctx, func, *args, **kw)
      File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 85, in callWithContext
        return func(*args,**kw)
    --- <exception caught here> ---
      File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
        why = selectable.doRead()
      File "/usr/lib/python2.7/dist-packages/twisted/internet/process.py", line 291, in doRead
        return fdesc.readFromFD(self.fd, self.dataReceived)
      File "/usr/lib/python2.7/dist-packages/twisted/internet/fdesc.py", line 94, in readFromFD
        callback(output)
      File "/usr/lib/python2.7/dist-packages/twisted/internet/process.py", line 295, in dataReceived
        self.proc.childDataReceived(self.name, data)
      File "/usr/lib/python2.7/dist-packages/twisted/internet/_posixstdio.py", line 77, in childDataReceived
        self.protocol.dataReceived(data)
      File "/usr/lib/python2.7/dist-packages/twisted/protocols/basic.py", line 571, in dataReceived
        why = self.lineReceived(line)
      File "/home/pouwelse/GITHUB/ipv8-dapps-loader/twisted/plugins/cli_plugin.py", line 134, in lineReceived
        self.dapp_menu_items[int(line)].values()[0](line)
      File "/home/pouwelse/GITHUB/ipv8-dapps-loader/twisted/plugins/cli_plugin.py", line 224, in download_dapp
        self.dapp_community.download_dapp(self.context['info_hash'])
    exceptions.KeyError: 'info_hash'

[mitchellolsthoorn]

Sprint Progress:

This sprint I ran into a couple of hurdles that slowed down my progress.

My main focus was on the following tasks:

• Cleaning up some code, to make it easier to work with, less likely to break
• Solve bug in dApp crawler
• Implementing all functionality through a REST API interface
• Working on a website to function as a GUI that could then be used in the Android App interface
• Building a twisted web server
• Cross-compiling libtorrent and libtorrent bindings
• Trying to compile the mp3 hack, incorporating libtorrent and adding all required communities and dependencies
• Injecting this mp3 in the Android app

These experienced hurdles were:

• Looking up a lot of information on how certain things work e.g. python for Android, twisted web server
• Having trouble with the compilation step
• REST cross-origin request protection

Completed tasks:

• Crawler bug
• REST API
• Basic website
• Twisted web server

Task still working on:

• Compiling mp3 with integrated libtorrent
• Solving Cross-origin policy
• Testing if all separate components work as a whole in the Android App

REST API:

Basic Website:

[synctext]

Meeting minutes:

• next sprint goal
  • make it work :-)
  • getting ready for Android "power-shell" release
  • demo examples: runtime addition of GUI elements, screens, requester prompts and popups
• decision point: continue the Android-track or document for thesis.tex and focus remainder on PC/Tribler?
• Future sprints
  • decentral market (orderbook + Bitcoin wallets)
  • decide: minimal trustchain app voting or emphasis on code review
  • Pound to Euro trading as goal?

[synctext]

Todays meeting minutes:

• Solid progress on Android
• Operational dynamic GUI, runtime additions, dynamic code importing
• decision point:
  • document Android plug-in
  • write thesis chapter: "problem description"
  • continue technical work
  • create plugin for trust
  • event processing and basic GUI
  • running code in experimental stage
  • combo of two above issues
• Future Sprint ideas:
  • trust plugin and event processing
  • select from multiple available trust functions
  • trust in code, code reviews and voting
  • Decentral market (orderbook + wallets)
  • Ripple functionality; Pound to Euro trading as goal?
  • transfer money using IBAN with public accountability
  • is the money transferred?
  • prevents lots of scams: "yes I just transferred the money to you"
  • zero-knowledge proof magic?

[synctext]

Storyline input: This thesis presents a framework and implementations of a truly universal binary.

[mitchellolsthoorn]

Sprint Progress

This sprint I focused on:

• Android:
  • Making the Android app a little more robust
  • Show a loading screen while the backend is loading
  • Auto refreshing the dApp interface
  • Shutting the IPV8 service down when the app is closed
• dApp loader:
  • Creating an event processing system that dApps can use to communicate through locally. This allows multiple smaller dApps to function as a bigger one
  • Fiddling with the web imports for the Android client
• Trust plugin:
  • Getting to know how it works
  • Creating a community around it
  • Making a trust dApp
  • Converting trust code from script to proper algorithm (In Progress)
• Thesis:
  • Working on the problem description chapter (In Progress)

Due to computer networking lab preparation, I had less time than I would have wanted. I really wanted to get a good start on writing the thesis, but this didn't work out in the end. I also had a hard time figuring out how to get the actual problem and subject in writing. I think this is an important discussion point, because getting a clearer picture of this would help me in writing the thesis.

Things I would like to work on in the next sprint:

• Thesis
• Conference presentation
• Trust in code

Android app:

[synctext]

Meeting notes: proposal for problem description. Risk is that it comes across as merely a plug-in system. dramatic like: "this thesis presents a multi-plaform binary and new paradigm for trustworthy computing. We build on the large body of work around smart contracts and make it more generic, scalable, removed global consensus, and need for oracles." etc. Compare Debian, NodeJS, Winamp plugins, Facebook plugins, and Dapps. Argue that this thesis represents the next step in the continued evolution of computing models.

Next steps:

• thesis latex: finish 1 chapter in draft.
• wrap up the Android multi-platform binary + GUI
• define your Trust API v0.1 for the dApp
• scale! Not the crawler, but the walker (1 double-signed record/ introduction-request).
• load balanced random walker (live edge is future)
• trust as a microservice!

[mitchellolsthoorn]

Sprint Progress

This sprint was not very productive. I wanted to get a lot more done, but I had to spend all my time on finishing the lab assignments for the computer networking course since otherwise, the lab would be in big troubles. I am now done with my part and transferring all the other tasks to different people, so I can focus more on my thesis.

I did work on a very rough beginning of my first chapter of the thesis and I am continuing with that for the next week before I am going to the conference

[mitchellolsthoorn]

Draft of outline for the story line: Master_Thesis.pdf

[synctext]

Nice read, three phases of reusable code

• object orientation
• component-based
• service oriented architecture (SOA) Plug-in ecosystem development
• Winamp, Facebook, XMBC plug-ins: rise of the amateurs, crowdsourcing
• Pro-only: Comprehensive Perl Archive Network from 1995 (24 years old, now essential dead)

Next sprint suggestions:

• small history of software reusability+user-extensible software, first Chapter 1 text draft
  • Now these are two different historical movements
  • This thesis will present a unified framework for developer modules and user plug-ins
  • Defining difference between Debian modules and NPM plug-package
  • Well-established culture of creating runtime RESTful APIs in NPM (not in Debian)
  • NPM is a micro-services community (not in Debian)
  • Developer selects a version of an NPM package (Debian solves versioning, NPM created a special kind of versioning hell)
• coding on dApp for trust-crawler (Crawl+GUI)?

Possible thesis final epic experiment:

• Several trust module implemented, same API (e.g. Reddis, Neo4j, and traditional SQLite) other trust algorithms
• User can switch between them, tune, and configure them.
• Future Fantasy feature: copy existing plug-in, edit the source code within Tribler, execute , and dApp publish.
• Deployment in Tribler ! ?? !
• Scalability of thesis dApp manager and entire ecosystem
• Permissionless software engineering, expose the source code, enable run-time code editing (beyond smart contracts execution model)
• enables live in-place updates for Tribler services! (live Linux kernel update patching)

[synctext]

Related work around running untrusted code in kernel space.

Berkeley Packet Filter (BPF) has turned into a broad Linux kernel mechanism for untrusted code. eBPF can run user-supplied programs inside of the Linux kernel, at ring 0 even. @qstokkink this might be inspiration for generic "IPv8 communities" and may unlock smarter smart contract by finally providing parallelism. It is especially designed for high-bandwidth stuff. Details:

• https://lwn.net/Articles/740157/
• https://blog.cloudflare.com/cloudflare-architecture-and-how-bpf-eats-the-world/
• HN discussion https://news.ycombinator.com/item?id=19947970:

[synctext]

Related work also having our features; https://www.kickscondor.com/on-dat/

"It’s clear that there are tremendous advantages here: Dat is apps without death. Because there is no server, it is simple to both seed an app (keep it going) and to copy it (re-centralize it). .. In fact, it now becomes HARD:IMPOSSIBLE to take down an app. There is no app store to shut things down. There is no central app to target. In minutes, it can be renamed, rehashed, reminified even (if needed)—reborn on the network."

[synctext]

Related work, mention in .tex thesis. Live code injection in Python: https://pyrasite.readthedocs.io/en/latest/GUI.html Extensive tooling.

[synctext]

Plugin malicious code attack: https://harry.garrood.me/blog/malicious-code-in-purescript-npm-installer/

[synctext]

Thesis progress meeting, state-of-code, discuss the defining thesis experiment&chapter. Outcome of brainstorm. It is possible to create several trust functions. dataset and existing code for putting in IPv8 dApps:

• https://github.com/pimotte/msc-experimental-code/blob/master/mcrep/netflow.py
• https://github.com/pimotte/msc-experimental-code/blob/master/mcrep/pimrank.py

Current feature includes a crawler. Ask neighbors for active gossip for all known dApps. Focus for next sprint:

• explore existing trust code
• try to get something up and running
• write a solid draft of a single thesis chapter (Android GUI experiments in IPv8 module, related work, problem description) Error (running with an old state-dir):

  2019-08-16T14:42:03+0200 [-] Unhandled Error
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 396, in startReactor
      self.config, oldstdout, oldstderr, self.profiler, reactor)
    File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 311, in runReactorWithLogging
      reactor.run()
    File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1243, in run
      self.mainLoop()
    File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1252, in mainLoop
      self.runUntilCurrent()
  --- <exception caught here> ---
    File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 878, in runUntilCurrent
      call.func(*call.args, **call.kw)
    File "/home/pouwelse/GITHUB/ipv8-dapps-loader/loader/community/dapp/community.py", line 388, in _check_votes_in_catalog
      creator = tx_dict[DAPP_BLOCK_TYPE_VOTE_KEY_CREATOR]  # type: bytes
  exceptions.KeyError: 'creator'

[synctext]

Related UMD work: This repository formalizes the design and implementation of the Universal Module Definition (UMD) API for JavaScript modules. These are modules which are capable of working everywhere, be it in the client, on the server or elsewhere.

[mitchellolsthoorn]

Progress and planning update:

Some progress was made, but not the amount that I was aiming for. This is mainly due to underestimating the time it took to prepare and actually move to Delft and me becoming ill.

A proof of concept was made to see how modules could be used to interchange logic that can change the output behavior of an application. I ran into some issues getting the two algorithms that we chose to run since the code was very old and outdated and without any documentation. Eventually, I managed to update all the outdated functions and figure out how to get it running.

I haven't managed to make any progress on the report size of the project.

To make the process more structured and focussed I have created the following planning: Week 37: Get better, Wrap up some last moving items, Expand the trust function modules proof-of-concept Week 38: Draft chapter Problem description and related work Week 39: Expand on code reviewability aspect of the project Week 40: Draft chapter Android experiment and possibly trust experiment Week 41: Draft chapter module and framework

[synctext]

Thesis outline discussion:

• intro on problem of code re-use, software reusability+user-extensible software
• problem description
  • we need a unified framework for developer modules and user plug-ins
  • beyond state-of-the-art requirements:
  • crowdsourcing of code
  • source code inspection
  • trust function
  • live overlay
  • dynamic loading
  • runtime-upgrades
  • developers communities around micro-services
  • owned by everybody and nobody, no central servers (self-governance)
  • show the viability of our ideas with a non-trivial use-case
• design and implementation
  • module discovery gossip
  • rating and voting
  • code screenshots
  • LoC + code coverage stats
• Experimentation and evaluation
  • trust explorer example
  • 9 fully functional modules
  • demonstrate the viability of run-time changes and alternative algorithmic approaches
  • open questions:
  • how do we evaluate this?
  • how do we show that it works as intended?
  • we successfully bypassed the Android security model with protection against dynamic code injection using Webview
  • pure technical (module loading times, memory usage impact, event-driven impact, event processing workload analysis, peak performance estimation)
  • Github/NodeJS/Debian replacement workload analysis ("github is the largest code host in the world, with 20 million users and more than 57 million repositories"

[mitchellolsthoorn]

Progress update

Worked on thesis the last week. Added draft of the introduction and the problem description. Had a hard time trying to separate the different sections: Introduction, Problem description, and related work.

Some personal notes:

• Should these be strictly separated or can they flow over into other sections?
• I am questioning when to introduce certain concepts
• How could I integrate parts of the paper in the thesis
• Should the design chapter follow chronological order or from overview to granular

Progress can be found in: https://github.com/mitchellolsthoorn/master-thesis

[synctext]

• add 1-3 sentences at the very start which explain the thesis title
• introduction has storytelling structure. Use a more related work focused structure. Provide citations for these bold claims.
• intro: mention + screenshot prior work that social trust can be used; DevID prototype
• Problem Description
  • devise a software re-usability framework
  • validate this with real-world complexity
  • you solve world software problem, no mention of Tribler yet
  • make broader
  • Tribler is merely an example / possible application
  • many people tried, proven to be a hard problem
  • trade-off re-usability and usability
  • we propose a new balance between the two.
  • yet another learning attempt
  • our unique requirements: ToDo into a Related Work table (red and green ticks)
• Design Principles (more scientific high level; not TFTP)
  • trusted execution through openness, crowdsourcing, and voting (e.g. Ethereum VM)
  • NodeJS == social trust
  • Github == social
  • Solidity == ?
  • Event-based architecture
  • Internet-based (live overlay)
  • any-platform (even including Android, GUI framework; thus only common denominator: HTML)
• Implementation and experimentation
  • architecture picture of crawler
  • evaluate event handling system?

[synctext]

Scientific thesis title (less engineering focus): "Implementation of a collective-intelligence coding framework"

[synctext]

Suggestion for problem description related work: Some theoretical grounding for coding frameworks exists in the realm of "enterprise software". Evolvability is the cardinal concept in "Normalized Systems Theory", which states that the competitive environment changes rapidly of organizations and therefore they must adapt themselves accordingly in an agile way. This leads to software development tools with versioning in data, APIs, and data processing. This leads to a system with minimal dependencies. We reproduced the following four principles from [NST] here:

– Separation of Concerns, stating that a processing
function can only contain a single concern (i.e., change
driver or each part which can independently change);
– Data version Transparency, stating that a data structure
that is passed through the interface of a processing
function needs to exhibit version transparency (i.e., not
impacting processing functions in case it is updated);
– Action version Transparency, stating that a processing
function that is called by another processing function
needs to exhibit version transparency (i.e., not impacting
its calling processing functions in case it is
updated);
– Separation of States, stating that the calling of a
processing function within another processing function
should exhibit state keeping (i.e., before calling another
processing function, a state should be kept).

plus please discuss Lehman law

[mitchellolsthoorn]

Progress update:

Back from conference in the states.

Remaining days I continued working on writing of the thesis. I still have to resolve points from the previous meeting notes, but I wanted to focus first on getting some more concepts on paper, so it is easier to see the bigger picture and the storyline. Afterward, I will start refactoring to correct parts that could be made more scientific and less storytelling.

I have worked on creating diagrams for different parts of the application, so they can be more easily explained. Especially activity diagrams would help with explaining the process for some of the decentral operations. I have started writing the Design and Experiment/Evaluation chapter and the section about the Android app experiment.

In the coming two weeks I want to focus on finishing the code and writing tests.

[mitchellolsthoorn]

Suggestion new title:

• Balancing re-usability and usability in modern software development.
• Next evolution of modularized code execution
• Balancing code evolution and code development accessibility

[mitchellolsthoorn]

The Lehman law matches the topic very well. I can definitely use that. Thank you.

The other resource you mentioned about the Normalized System Theory seems to describe the development of highly abstracted and engineered software application that allow for agile development, without breaking and with minimal change. I don't know if this is very suitable

[synctext]

please mention: live code updates: https://github.com/julvo/reloading

[mitchellolsthoorn]

Need one more week to finish the final product and experiment. After that focus completely on adding remaining concepts and rewriting and refining the report.

Had trouble finding good metrics for the experiment that won't be very subjective or that depend on the composition of the network. Technical metrics are not suited best for comparison with other platforms.

Design chapter structure/notes:

• Modules search for dependencies downwards
  • View layer triggers search for logic layer
  • Logic layer triggers search for other logic layer
  • After search they register with the higher module
  • Module abstracts classes for registration mechanism
• Module distribution mechanism
  • Send message to all neighbours when voting
  • Send message to random neighbour every x time units to keep modules circulating
  • Random sending based on votes, lifetime, reputation
  • Each nodes downloads module to increase availability in the network
  • After x time units unused/unstored modules get deleted
  • Mention malicious nodes
  • Track of module is used to determine if it can be deleted
  • Find different module versions by module identifier
• Modules
  • Modules should have votes per version to prevent major code changes
  • Code should be viewable
  • Version number is hash code to prevent version tempering
  • Each module has own pub key for identifier
  • Decide to keep author in module identifier
  • Decide if each version should be a separate module, has downsides to upgrade speed
  • Key value store for storing settings in dependent modules
• Execution
  • Docker runner
  • Local runner
• Experiment
  • Distribution spread factor
  • New node Discovery time/factor
  • Module creation speed
  • Speed to distribute through network with instant voting
  • Speed to distribute through network for update
  • Bandwidth usage (hard to measure)
• Questions
  • Why Python? Because dynamic?
  • Why IPv8
  • Is it more usable?
• Motivation
  • Another attempt at holy Grail of software reusability

[synctext]

Quote @mitchellolsthoorn : Normalized System Theory == purist, holy grain and therefore inefficiency beyond usage. Too complex. Without simplicity all agility is lost.

Python does not need to be motivated. All engineering details are merely documented, not meticulously defended. "We use Trustchain as the distributed ledger in our experimental implementation."

IPv8 runtime model versus Ethereum fundamental brokenness (e.g. scalability and cost). The Ethereum model is not long-term sustainable:

Back in October’17, an investor sent 1,700 ETH to a contract (AirSwapDEX) with a gas price
of 400,000 Gwei and gas limit of 592,379. The Tx failed for some odd reason but the
investor was charged whopping 236 ETH ($122,086 as per today’s price)

"Trustchain does not have a native cybercurrency such as Ethereum, it provides a transaction recording ledger". Plus distributed trackers are currently still a central dependency (e.g. 25MByte bootstrap is still ongoing work).

Your thesis work requires a name for ease of referencing. Pasta-Frame: the next evolution of modularized code execution. Key property is permissionless code execution at near-zero cost. The code execution architecture defines the maximum complexity of the code that can be produced. JVM, Nodejs, CPAN perl modules and other real-world framework and the connection between modules determine the Maximum Complexity of Applications (MCA) which a single company, an global consortium or open source community can create. We devised the first architecture to take the MCA as the cardinal design optimisation. Science what defines MCA? What constrains/boosts MCA? The interconnection fabric is the key determinant for the MCA. How data flows between modules, how the future-proofing is arranges, how any piece of code can interconnect with any other code, and how can we devise the universal module interconnector?

Feel free to cite this 2003 work on dynamic configuration and dynamic runtime loading of code modules with a dependency graph with running code:

General remark: please always keep the science first and omit engineering details. Runs!

[synctext]

"Dependency managers now exist for essentially every programming language: Maven Central (Java), NuGet (.NET), Packagist (PHP), PyPI (Python), and RubyGems (Ruby) each host more than 100,000 packages. The arrival of this kind of fine-grained, widespread software reuse is one of the most consequential shifts in software development over the past two decades. And if we're not more careful, it will lead to serious problems." https://queue.acm.org/detail.cfm?id=3344149

[mitchellolsthoorn]

I continued working on the report. Although, the written isn't going as fast as I want. I struggled a lot during the last sprint as I am not making the progress I want to. I have rewritten the structure of the Introduction and moved parts of the structure around. I want to discuss if this new structure is a better fit for the thesis:

Introduction:

• Fancy introduction text to introduce the title and captivate the reader
• Explain what is already done by other people
• Explanation of what happened in the industry over the last couple of years
• Explanation of concepts used in the thesis
• Explain what is missing from the other research and what the thesis tries to solve
• Explain the structure of research/thesis Problem description:
• Explain in more details the problems that were left out in the current research and why these are important
• Explain scope of project Design:
• Explain the proposed solution to the mentioned problems through the framework

[synctext]

report (25).pdf review of current thesis:

• Vage "How to minimize the risk associated with the use of dependencies?". More concrete 'can we use social trust and crowdsourcing to improve security of libraries'?
• consider adding: "non-revocable publication of libraries"
• "It tries to find a balance between the software practices of Today and the impractical concepts of the future.", reformulate the future
• Chapter 2: "This work sets out to create a unified framework for reusable developer modules and user plug-ins built on top of IPv8."; make a clear opening sentence. Tribler comes later chapter: 'We use a concrete use-case driven methodology to advance the cause of software re-usability. '.
• live overlay: or integrated autonomous dissemination. No central entity or central governance, self-organising. No owner as person or organisation
• trustworthy code
  • open ecosystem
  • crowsourcing
  • trust function
• runtime engine
  • dynamic loading
  • integrated autonomous dissemination
  • seamless upgrading

ToDo: animated .GIF of whole trustchain .APK starting, loading, module fetching, trustchain starting, crawling, and browsing ? Use as leading example in thesis experiment? Merge Chapter 5 in other chapter?

[mitchellolsthoorn]

Progress update:

• Rewrote majority of introduction to be more polished
• Changed structure of problem description and added more information

[synctext]

LATEST raw .pdf of master thesis

• Re-write the 2.4. Runtime Engine more as a problem description, instead of a solution direction. Like: "A key bottleneck for re-usability and usability is the lack of support within the execution environment. Requirements for our envisioned runtime engine are as follows...". Requirement: "runtime support"?
• "Euthereum", modern spelling checker?
• Chapter "FBASE design"

[mitchellolsthoorn]

Progress update

• Fixed some points in the Problem description
• Rewrote the FBase design chapter
• Filled in the Final thesis document
• Want to have review version coming Friday

PDF

[synctext]

Comments:

• "3.5. Blockchain Storage"; organisational principles?