search

MetacoSA / NBitcoin

Comprehensive Bitcoin library for the .NET framework.
MIT License
1.87k stars 846 forks source link

Portable library missing parts #56

Closed fonix232 closed 9 years ago

fonix232 commented 9 years ago

I've decided to collect all missing parts and to-dos into one big issue so that it can be solved step-by-step.

NicolasDorier commented 9 years ago

I commented on https://github.com/NicolasDorier/NBitcoin/pull/55, I am curious about why you want to port protocol when socket not available.

NicolasDorier commented 9 years ago

For the PaymentRequest, it also seems that X509Certificate2 is only supported on windows, making it hardly portable to other plateforms. I guess, I can manage to make BC handle it if we need to though.

fonix232 commented 9 years ago

I suppose that would be the best way to go, then. BC has been successful (to an extent) to provide black-box APIs on most platforms, so using it might be the only option.

For now, though, my goal was to be able to work with some tools - and the rest of the additions, namely #55, was just a "eh, why not".

NicolasDorier commented 9 years ago

Just released the 1.0.7.6 (975e92c12b90e972b64cd6db4795a46f2f76fa8c)

fonix232 commented 9 years ago

Awesome work!

fonix232 commented 9 years ago

Nicolas, just to confirm - on WinRT (Profile111), it is not possible at all to create a truly networked app for now? All other things (creating transactions, signing them, etc.) works, but those need an external service to communicate with the Bitcoin network?

NicolasDorier commented 9 years ago

that is right... funny thing is that the networked features can be included in windows phone since it supports socket, but not winrt.

I wanted to include it to windows phone this week, but my visual studio won't let me create a phone project with windows 7.... (it worked in the past) I guess I'll have to wait windows 10 :(

fonix232 commented 9 years ago

Only Windows Phone silverlight profiles (WP8.0 and 8.1 Silverlight), which are going obsolete with Win10.

However I do not understand why MS is discarding the Berkeley Socket type in general - it is a huge part of many applications and moving to the WinSocks base is a pain in the arse.

-----Original Message----- From: "Nicolas Dorier" notifications@github.com Sent: ‎2014.‎12.‎16. 17:41 To: "NicolasDorier/NBitcoin" NBitcoin@noreply.github.com Cc: "fonix232" fonix232@gmail.com Subject: Re: [NBitcoin] Portable library missing parts (#56)

that is right... funny thing is that the networked features can be included in windows phone since it supports socket, but not winrt. — Reply to this email directly or view it on GitHub.=

NicolasDorier commented 9 years ago

Really ? do you have source ? To my understanding what they call "universal app" is nothing but portable profile spanning winrt, win8 and 8.1 SL.

For the socket, I think the reason is they the "Async it all" pattern everywhere.

But, I'm not sure it is a big problem to have a dumb client relying on a web service instead of direct connect to the network. It hurts privacy, but an app can mitigate that by making their API public and allowing to change the service's url in the app.

fonix232 commented 9 years ago

Well it was stated multiple times that Win10 will bring all platforms into one (OneCore), having the same APIs, et cetera. Actually I'm testing Windows 10 right now (with Cortana!) and it is indeed bringing phone, tablet, winRT and desktop together.

NicolasDorier commented 9 years ago

Interesting video on channel9

Their new API is WinRT based, so don't count on a Socket class :(

fonix232 commented 9 years ago

I just had the rad-est idea!

It is more or less your implemented central-server approach, but still slightly different... Why not use a public service on unsupported platforms? Say, block.io - open API, free to use, unlimited calls. Implement that, on unsupported platforms prompt for an API key (they should register themselves...), then let their service handle the network transactions.

I think that would work.

NicolasDorier commented 9 years ago

Definitively and idea, but such library should be on separate projects I think, not into NBitcoin, but inside specific projects specific to the API provider.

Or are you talking about an API that would depends on a server that would itself depends on a service like block.io ? I think this should too be in separate assembly. By the way, I developed an Indexer highly coupled with Azure and NBitcoin. (open source, supporting colored coins, and wallet creation)

It is not highly documented for now though, I will make an API on top of that, but I will not release the web controller layer in open source. Once I've done that, the client API to this service will be open source and portable.

Block.io is not totally free and unlimited, there are some threshold are easily reached depending on what you want to do

fonix232 commented 9 years ago

Well, in theory it should be possible to create an Electrum server-like network. Centralized controller servers for storing the blockchain and serving clients information about a specific address, and sending&receiving the BTC network packets (transaction handling). Then the client-side is not so heavily loaded.

Of course this is just a theory, but I might start writing something soon.

NicolasDorier commented 9 years ago

Definitively. You can also make a SPV with partial merkel trees, so the client don't have to trust the server.

Basically, when the server find a transaction of interest, it would forward it to the client (polling or web sockets) along with the partial merkel tree of the block where the transaction appear.

realindiahotel commented 9 years ago

Hello, the idea that Windows Universal app is WinRT + WP 8 and WP8.1 SL is incorrect. What has actually happened is they created a WinPhoneRTwhich covers about 80% of the WinRT API. So when you target universal app you are actually targeting about 80% of WinRT. You can however of course add your platform specific code into each section, just the common code has to fit in that 80% footprint. SL is a different beast that aa mentioned will likely be done away with in Windows 10. The biggest problem apart from sockets that I forsee is background processing. I think it will be very difficult to try say listen for transactions due to the background restriction. To be honest, anything RT should really rely on a full blown blockchain parser in Azure firing notifications etc back to client. This way you avoid bloom filters etc as well. :)

NicolasDorier commented 9 years ago

Thanks for the precision

How is it different from portable lib 4.6 ? (http://www.microsoft.com/en-us/download/details.aspx?id=40727)

I think this portable lib the "80%" you are talking about isn'it ?

realindiahotel commented 9 years ago

Nah, PCLs (Portable Class libraries) were around before Universal Apps. They served as a common ground between WP SL and WinRT so I think you will find PCLs are actually even more bare bones than the 80% WinRT footprint of the universal app shared library.

realindiahotel commented 9 years ago

So in an attempt to clarify my half pissed and typing response, basically PCLs came before the 80% footprint. They can be used in Universal apps because they are less than the 80% footprint, you can't do a hell of a lot in a PCL and it was/is infuriating. Basically you are supposed to make interfaces or whatever and then use them in native implementation. It's a terrible way of doing things because you're forever reinventing the wheel between platforms. The universal apps os the first step away from PCLs and when we get true unification between WinRT and WinPhoneRT then we will do away with PCLs as we know them now and it will be back to good ol' class libraries like the old days :D

realindiahotel commented 9 years ago

Oh and yes you can now set a PCL to target the Universal App 80% footprint and it can be used in WinRT and WinPhoneRT, but if you do that it CANNOT be used in either WP 8 or WP 8.1 Silverlight. I hope I make sense.

NicolasDorier commented 9 years ago

Thanks for your details,

PCL is still very good for developing cross plateform, and nuget plays great with it. They are just "compilation target", not necessarily executing on the target runtime. And since most developer doing mobile stuff need to support more than windows, it is here to stay.

Universal App seems to be more than a common compilation target, it also is a common implementation. But such thing could be modeled into a PCL.

By the way, would you know the target name in Nuget for targetting a package with Universal App ? Can I do that from my win7 machine, or stuck at waiting win 10 ?

realindiahotel commented 9 years ago

Yes you are right, I of course forget about Xamarin and Microsofts push to bring .NET to other devices and I think only about Windows Store projects.

Yes so I should have been more clear, Universal App projects actually create two projects and a "shared" core. So you have a WinRT project, a WinPhoneRT project, and a shared core project (basically a PCL set to target the 80% footprint between the RTs) and you CANNOT use any Silverlight functionality in these projects. There is in fact still functionality that is only possible in a WP Silverlight app that can't be done in universal app.

Yes you are right about the compiler targeting, it just means each time you retarget you change what framework functionality you have available, so if you retarget a Windows 8.1 and Windows Phone 8.1 PCL to a Windows 8.1 and Windows Phone 8.1 Silverlight PCL, you get only framework available common to Windows 8.1 (RT) and Silverlight which is not much at all.

No you need the Modern Shell for universal apps to build/test and deploy the Windows 8.1 part of the universal app. So anything below Windows 8.1 will not be able to create universal projects. You will need to wait for 10 if you wont use 8.1

NicolasDorier commented 9 years ago

Ok cool. So I'll wait windows 10. I want ot make a package specific for universal app, thanks for the precision !

realindiahotel commented 9 years ago

Yes so to make package specific for universal app you would create a PCL and target "Windows 8.1" and "Windows Phone 8.1" and nothing else. Avoid anything with the word silverlight. My packages at the moment INCLUDE silverlight, although on hindsight I think that was dumb of me to do that, I am a WP Silverlight developer but it's time for me to smell the cheese and focus all in on Universal now I think, I think Silverlight needs to die.

NicolasDorier commented 9 years ago

The fun thing is that microsoft has the ability to trash so much of the product we love that we just got used to it :D

I was pissed of by Silverlight (web) story, then XNA, but now I take it relatively easy (and so you). I guess we are getting used to it... ;)

realindiahotel commented 9 years ago

Tell me bout it. I clung on to .NET 1/2 and WinForms for dear life. WPF came along amd I didn't want anything to do with it haha. Then WP8 came and so that was my first big venture into Silverlight and they are killing it anyway.

Although, to be fair, they are learning (and I'd say being dragged through the coals by people like yourself who have to keep changing) and at least this time the are also including standards like HTML5 and JavaScript going forward, so with all the open sourcing and the embracing of industry standards as opposed to proprietary, I think think this rapid rate of change will quiet down. This new fella Nadella or whatsis name I think he is the best thing since sliced bread for Microsoft.

fonix232 commented 9 years ago

Microsoft is heading in the right direction nonetheless.

However I still do not understand the reason to exclude Berkeley Sockets from WinRT APIs. There IS, however, some implementations of the generic Socket class for WinRT.

It should also be possible to write the generic socket from the grounds up, WinRT compatible, as a facade over the WinSocket implementations. A guy already done so, to an extent, see here: https://winrtsock.codeplex.com/ but I suppose it is not yet complete.

NicolasDorier commented 9 years ago

yes, I am too once again loving microsoft ;)

My guess for socket is that Microsoft might not want long lived TCP connection on winrt. Such connection would be messed up in case of tombstoning. Since HTTP does not depends on the TCP session, there is not the same problem that another protocol might have. In fact Bitcoin depends on the TCP session. If the session break, you need to make a new handshake. But one can point out that HTTPS is the same...

Just a speculation.

fonix232 commented 9 years ago

So Nicolas, do you think that that implementation could be used?

Although it is not exactly a portable non-language-specific library, I think it could be ported with relative ease.

NicolasDorier commented 9 years ago

What, the berkeley socket implementation from the winrt version ? I'm not convinced about the "ported with relative ease".

And even if it is, I'm not sure we can even make an app play nicely with tombstoning events from microsoft plateforms. I think it is easier to just keep the devices dumb, and use a web service that do the heavy lifting. Also, we should not forget that NBitcoin allows you to speak with the network, but is not a full node implementation. It does not validate anything, it just handle the communication part of the protocol. That's why I'm actually using it only connect with a trusted local node. (even if in theory you can make a full node out of it)

I am working on such service. The code will not be open source though. (Using NBitcoin.Indexer under the hood, which is open source) Take a look at http://rapidbase-test.azurewebsites.net/

realindiahotel commented 9 years ago

100% agree with you Nicholas. As you say, tombstoning and the short time frame and intervals for background tasks makes running a full node on WinRT/WinPhoneRT impossible. As Nicholas says, they best thing would be to have a full node in the cloud and devices can register addresses and receive notifications from the cloud node whilst keeping private keys on device. Like Nicholas I am going to have a go at achieving this.

NicolasDorier commented 9 years ago

Thashiznets, you might be interested by my other project "NBitcoin.Indexer". It use azure table as a backend. I'm building on top of it.

The user API is not stable these days since I accomodate a lot (using it in 3 different customer project). However, the indexing works well and is stable. If you think using it I advice you to use your own forked version for now. (so you don't suffer from my api changes)

This is what I am using for http://rapidbase-test.azurewebsites.net/

realindiahotel commented 9 years ago

@NicolasDorier yea nice one! I'll have a look, I want to write my own code because that way I can easily maintain it when protocol changes etc, I'm not very good at deciphering other peoples code haha! But definitely that kind of thing you are doing is the future. Big cloud node clusters serving devices is the way it needs to be, I believe full node P2P rates are dropping off due to the implementation od SPV clients using bloom filtering etc.

NicolasDorier commented 9 years ago

Ok, if you want to do it, for making a fast and scalable storage here is what I've done. (works very well) When I index a transaction, I don't put in the same db row the previous txouts. It makes indexing quick.

However, on query time, I fetch the previous txout and store it in the same line, so that later on a request will take 1 database call. (Lazy Loading)

When I index balance movement (wallet or address), I store it so the natural order is Height descending. Data being : height, txid, blockid, coinspent,coinreceived. Same thing, I don't store the spent coin in the line. I do it at query time lazily. When I request the balance on an interval [from height 300 000 to 290 000] for example, then eventually 1 query bring me back all the balance change.

Also, the client is responsible for filtering the orphaned balance line, not the indexer. This make the indexer easily scalable on several machine without the risk of incoherent blockchains.

NicolasDorier commented 9 years ago

I don't intend to implement BIP70 in portable library. As I have stated in the bitcoin dev mailing list, asking to a mobile/tablet developer to implement a cross plateform and secure certification chain verification is not reasonable. (and something NBitcoin can't do on its behalf transparently) I recommend to forward the PaymentRequest to a trusted web service that will do the verification on the behalf of the device. NBitcoin still support BIP70 for windows plateform (depends on X509Certificate + chain verification done by the OS)

Search for "[Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?" in the mailing list to see my points.

realindiahotel commented 9 years ago

@nicolasdorier where can I find the mailing list? Isn't BIP70 a method of creating a payment request for use in QR? Is your concern a man in the middle that might send a malicious payment request in place of the legit one? (valid concern)

NicolasDorier commented 9 years ago

You can see here.

My point is by using BIP70, you put the burden of certificate verification on the shoulder of the developer.

So the developer has 3 choices :

Also protobuf is not supported perfectly on all plateforms, unlike JSON.

My point is that payment requests should be simple JSON document over HTTPS, so the plateform take care of the validation and not the developer. Garzik and Mike point was that with BIP70, devices like Trezor don't need a full SSL layer... True but mobile users have one, so BIP70 is not well adapted for mobile/tablet cross plateform development.

Then they responded me with some bullshit about JSON exploit, and how protobuf was better used by google blabla, which was not my point. :/

NicolasDorier commented 9 years ago

I am closing this PR, I implemented all I could.

UPNP is removed from NBitcoin, I don't plan to make BIP70 portable for now, for the reason I enumerated.