Error writing block to datastore: Access is denied.

[jaynagpaul]

Fixed by ipfs/go-ds-flatfs#34, needs to be bubbled up. ~ Kubuxu

---

Version information:

go-ipfs version: 0.4.9- Repo version: 5 System version: amd64/windows Golang version: go1.8.1

Type: Bug

Severity: Medium

Description:

Started ipfs daemon: ipfs daemon Runs for a bit and connects to peers. Starts throwing errors in the console. 16:48:35.891 ERROR  bitswap: Error writing block to datastore: Access is denied. bitswap.go:322 and 16:48:39.904 ERROR commands/h: err: write tcp4 127.0.0.1:5001->127.0.0.1:56762: wsasend: An established connection was aborted by the software in your host machine. handler.go:288

[adi-dev]

is there any progress on it, still happening on 0.4.10, similar setup

[wolfsgrove]

Same problem here.

Agent Version go-ipfs/0.4.10/ Protocol Version ipfs/0.1.0

Win 10 amd64 binary install

[leerspace]

I also have been getting these errors every once in a while, and was finally able to reproduce it while capturing --debug output from the daemon. At the time of the "error writing block to datastore..." there is also an ACCESS DENIED error at that time in procmon. I'm also seeing quite a bit of activity from Windows' SearchIndexer.exe, SearchProtocolHandler.exe, and MsMpEng messing with files in .ipfs\blocks\. More importantly, I see them messing with the same file when ipfs is also trying to use it.

In the example of the error on my windows machine it was related to the following file:

C:\Users\John\.ipfs\blocks\LU\put-675545035

I'm going to disable Windows' file indexing on the .ipfs directory and try later to see if I still get this. If I do I'll add an exclusion to Windows Defender for the .ipfs directory too.

[leerspace]

FWIW since turning off Windows indexing on the .ipfs directory I haven't encountered this error again. I haven't done anything with antivirus.

@jaynagpaul @khokkanen @adi-dev I'd be curious if turning off indexing for the .ipfs directory also works for you.

[Kubuxu]

It would make sense, when we are writing to some key we first create temporary file and then move it under correct key. If windows runs search indexer and it accesses the file between those two happen and is still accessing the file when we are trying to move it it will fail.

[leerspace]

I just moved my .ipfs directory from my small OS SSD onto a larger HDD and started getting this error again. I thought this might be related to the hash I was pinning containing some virus content (the textfiles.com archive from archives.ipfs.io), but I still have some examples even after excluding the directory from Windows Defender. FWIW for that archiving I was also getting some "Error writing block to datastore: Operation did not complete successfully because the file contains a virus or potentially unwanted software." messages, but those went away when I excluded the .ipfs directory (not necessarily recommended).

I got an example in procmon and it looks like two different threads under the ipfs.exe process are attempting to rename put-* files to the same *.data files before the first thread has closed the file. Maybe there's some kind of race condition exacerbated by having the datastore on a relatively slow disk (?).

Disclaimer in case it's relevant: when moving my .ipfs directory to the HDD I created a symbolic link in the original location, so IPFS is accessing it through a symbolic link.

[djdv]

FWIW I get this error seemingly randomly, I have both indexing and defender completely off on my main machine, anything that can exclude the .ipfs directory is set up to ignore it as well.

[PCSmith]

This is making the system un-usable for me. Thousands of these access denied errors... Most attempts to access files fail with something like this shown in the browser:

When browsing to: http://localhost:8080/ipfs/QmZpc3HvfjEXvLWGQPWbHk3AjD5j8NEN4gmFN8Jmrd5g83/cs/bob.jpg

"ipfs cat /ipfs/QmZpc3HvfjEXvLWGQPWbHk3AjD5j8NEN4gmFN8Jmrd5g83/cs/bob.jpg: Failed to get block for QmdrCNKCexqTz8cW1PVc3jSQNCNU4NGwEL9iC85KAM8NYc: open C:\Users****.ipfs\blocks\3F\CIQOM4HRVP7HYYMWM5WPMR6LYQGV3VNTVEMSKAFO6OO27356ZWYS3FI.data: The process cannot access the file because it is being used by another process."

Refresh a few times and it works.

This should be rated critical as it's a show stopper for usage.

[leerspace]

@PCSmith mostly out of curiosity, have you disabled Windows indexing on the .ipfs directory? Is the .ipfs directory on an SSD or HDD?

[whyrusleeping]

@PCSmith do you have an easy way to reproduce the error? Is it just "run ipfs on windows?"

[leerspace]

@whyrusleeping I know I'm not @PCSmith, but I can readily reproduce this error on a fresh windows 10 VM using the following steps to reproduce this error. This VM is running on a cheap consumer HDD if it matters.

1. install ipfs, ipfs init, then ipfs daemon in a cmd window
2. disable indexing for the .ipfs directory: right click > Properties > Advanced... > Uncheck "Allow files in this folder to have contents indexed..."
3. ipfs get Qmb8wsGZNXt5VXZh1pEmYynjB6Euqpq3HYyeAdw2vScTkQ in another cmd window (this is the XKCD archive)
4. try loading various albums in the browser from the Project Apollo Archive http://127.0.0.1:8080/ipfs/QmSnuWmxptJZdLJpKRarxBMS2Ju2oANVrgbr2xWbie9b2D/frontend/frontend.html

[PCSmith]

leerspace basically covers it.

I tried it a few different ways with no changes to the issues. Complete delete re-init each time.

1. HDD
2. SSD
3. symlinked to a network drive iSCSI
4. indexed and non-indexed
5. anti-virus and none

I'm windows 10 (latest release). 64 GB RAM w/ high end proc.

[whyrusleeping]

Great, I just set up a Windows 10 VM. I will be taking a look soon.

On Wed, Aug 30, 2017, 8:27 PM Patrick notifications@github.com wrote:

leerspace basically covers it.

I tried it a few different ways with no changes to the issues. Complete delete re-init each time.

1. HDD
2. SSD
3. symlinked to a network drive iSCSI
4. indexed and non-indexed
5. anti-virus and none

I'm windows 10 (latest release). 64 GB RAM w/ high end proc.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ipfs/go-ipfs/issues/3971#issuecomment-326179284, or mute the thread https://github.com/notifications/unsubscribe-auth/ABL4HPuSomT0IgjTNVfXDSLl3EgIzr8Sks5sdigGgaJpZM4N2gba .

[hoffmabc]

Seeing this on OpenBazaar as well with fresh Windows VMs.

[PCSmith]

Any luck duplicating it on your end?

[whyrusleeping]

Hrm... I was unable to reproduce the error. Could be my ssd is too fast?

[hoffmabc]

Not sure. It might not be an issue on SSD drives but can't confirm. We added a looping hack to our code to make sure it goes through, but it's well..hacky.

[whyrusleeping]

This should not be a problem if you use badger, so can someone try out latest master to confirm? (ipfs init —profile=badgerds)

On Sep 11, 2017, 4:18 PM -0400, Brian Hoffman notifications@github.com, wrote:

Not sure. It might not be an issue on SSD drives but can't confirm. We added a looping hack to our code to make sure it goes through, but it's well..hacky. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[Stebalien]

Related issue from rust land: https://github.com/rust-lang/rust/issues/29497

We could set FILE_ATTRIBUTE_NOT_CONTENT_INDEXED to disable indexing but go doesn't make it easy to make syscalls like that...

[leerspace]

I tried building current master from source using the manual steps and then the cygwin way using this documentation, but I ran into errors I didn't know how to fix in both cases. The manual way ends up throwing a bunch of messages that look like this:

..\..\vendor\gx\ipfs\QmQUmDr1DMDDy6KMSsJuyV9nVD7dJZ9iWxXESQWPvte2NP\go-libp2p-swarm\swarm.go:36:2: cannot find package "github.com/whyrusleeping/multiaddr-filter" in any of:
        C:\Users\John\go\src\github.com\ipfs\go-ipfs\vendor\github.com\whyrusleeping\multiaddr-filter (vendor tree)
        C:\Go\src\github.com\whyrusleeping\multiaddr-filter (from $GOROOT)
        C:\Users\John\go\src\github.com\whyrusleeping\multiaddr-filter (from $GOPATH)

And like this using the cygwin way (after having to run dos2unix on multiple files that make install didn't like):

error: couldn't download with ftp (2)
Unable to download https://ipfs.io/ipfs/QmR27Do9gqx9VmuQTEX1UGXETSWYJTQzPzxS5FNUnySCv1/gx/v0.12.0/gx_v0.12.0_windows-amd64.zip. exiting.
make: *** [bin/Rules.mk:19: bin/gx-v0.12.0] Error 1

If someone has a current windows build of master I'd be happy to test. Or if there are updated build instructions for Windows that I can get to work I can also test.

[whyrusleeping]

@leerspace you should be able to do:

go get github.com/whyrusleeping/gx
go get github.com/whyrusleeping/gx-go
gx install
go install ./cmd/ipfs

[kpcyrd]

@leerspace the error of the second block looks like it's from dist_get: https://github.com/ipfs/go-ipfs/blob/c14a995fab81f894371ed06ec989c32bdd628e3d/bin/dist_get#L46-L52 Make sure you have either wget or curl installed and in your path.

[leerspace]

@whyrusleeping @kpcyrd thank you both for the help. While neither suggestion got me all of the way there, it at least got me past a couple of hurdles where the subsequent errors were relatively easy to solve (though there are a few things I'm not sure how I got past since I didn't keep track of everything I tried). Both of your posts make it sound like this should not be nearly as difficult as it was for me, so maybe I'm just doing something wrong or our environments are different.

The good news is that I have not been able to reproduce the "access denied" error while using the badger datastore (ipfs init --profile=badgerds)! :)

If anyone else who was getting the error wants to test this and is having trouble building this on windows, I'll happily share the executable I used to test. I'm just not sure whether it's good form to post a link to the binary in this issue.

ipfs version 0.4.11-dev-c14a995

[piravp]

Anyone has a solution for this yet? I tried running ipfs daemon, but got these errors trying to access the webui: 02:35:37.193 ERROR bitswap: Error writing block to datastore: Access is denied. bitswap.go:321 and then a blue screen. I'm running: Version v0.4.10 for windows 64bit on Windows 10.

[leerspace]

I don't know if anyone else has tested it, but using badgerds instead of leveldb for the datastore seems to fix it. There's an issue with badger on Windows in 0.4.11-rc2 that prevents badgerds from working on Windows, but it's been fixed in badger and the changes just need to make it up to IPFS.

The blue screen sounds like a separate issue. I haven't seen one of those in a long time (and never from IPFS).

[sandersaares]

I confirm issue remains with version go-ipfs_v0.4.11_windows-amd64.zip.

The badgerdb option, whatever that does, also fails:

I:\>ipfs init --profile=badgerds
initializing IPFS node at I:\Data
generating 2048-bit RSA keypair...done
peer identity: QmbMd4xxSWHQYPWNiMTxp37U3GDTXvW7C4vMkTGGwwrYWL
Error: Unable to open value log: Unable to mmap value log file: Not enough storage is available to process this command.

This defect is a complete blocker for my use of IPFS.

[djdv]

@whyrusleeping Is there any plans to implement migration methods that allow you to move from 1 datastore type to another? In this case leveldb -> badger. If not I assume we can just run 2 instances at once with different datastore paths, copying pins from the leveldb instance to the badger one.

@sandersaares This issue has been resolved upstream https://github.com/dgraph-io/badger/commit/273f40c6c24e1b188b68aeba1577de15e53fc015 I think the ipfs team is waiting for the badger api to become stable before updating though. https://github.com/dgraph-io/badger/issues/161

[magik6k]

@djdv For moving data there is https://github.com/ipfs/ipfs-ds-convert, though it currently has problems with moving data to badger, see https://github.com/ipfs/ipfs-ds-convert/issues/4.

The badger fix for windows happened before badger 0.8, so we can pull it in.

[leerspace]

ipfs/go-ipfs#4242 is related to the badger mmap value log error on Windows and can probably also be closed once badger is updated

[leerspace]

For anyone else who was anxious to use badger on Windows, the badgerds option is now working again on Windows (at least as of ipfs version 0.4.12-dev-1e73c3f) and the ipfs-ds-convert utility's previously referenced issue is now fixed.

[whyrusleeping]

Though note: badger support in ipfs is experimental. We can't make any guarantees about it yet. That said, please do try things out and give us as much feedback as you can, try to break it :)

[dang0ld]

I had this problem and was able to solve it.. mostly .. by opening up the daemon inside an Administrator Command Prompt instead of a user one.

This eliminated all the webui/permissions-related errors I was seeing in the console output - but now I am seeing other ones when actually just running the daemon. Will report back if I figure out anything else..

[kiahmed]

running on admin mode in win 10 pro, still get this error 20:10:29.518 ERROR  bitswap: Error writing block to datastore: Access is denied. bitswap.go:317

[avatar-lavventura]

I am having same bug on linux as well.

[Kubuxu]

@avatar-lavventura the cause for that will be competently different.

[Stebalien]

Fixed by https://github.com/ipfs/go-ds-flatfs/pull/34, needs to be bubbled up.

[dragon9783]

I got same issue on win 10, and ipfs version 0.4.15

08:43:17.459 ERROR    bitswap: Error writing block to datastore: rename C:\Users\win\.ipfs\blocks\2L\put-355713921 C:\Users\win\.ipfs\blocks\2L\CIQCYDJ23XAVLSZMK4JDHQK2TCHN6ASKTWAKP6VDP5D7LW5Z54KV2LA.data: Access is denied. bitswap.go:385

i change directory to full access control, it does not work

[Stebalien]

Crap... well, let's move this to a new issue as this one has definitely been fixed.