search

sql-js / sql.js

A javascript library to run SQLite on the web.
http://sql.js.org
Other
12.52k stars 1.05k forks source link

IndexedDB and concurrency support #124

Open alban-gerome opened 8 years ago

alban-gerome commented 8 years ago

Hi Kripken,

I am working on a web crawler and I need a database to store the links I find and avoid crawling the same link twice. I use PhantomJS 2.0 and this is supposed to support IndexedDB but I could not get a single code example to work so I am considering using your project instead. Have you tried IndexedDB in PhantomJS yourself?

Now to make the crawler faster I could have several instances of PhantomJS and split the workload of pages to scan. However I would have a single database for all PhantomJS instances. I was wondering whether your project supported concurrency methods.

Thanks,

Alban

kripken commented 8 years ago

I'm not familiar with PhantomJS, sorry.

sql.js is basically just a JS library, so you could run several of them in parallel in different web workers, to get concurrency.

alban-gerome commented 8 years ago

I mean that when one instance is trying the write another might be trying to read the same database table. That could cause issues.

PhantomJS is a headless testing browser using the WebKit engine, it does not display the page but it executes the Javascript. So basically you could use IndexedDB as replacement for sql.js except that it has not worked for me yet.

kripken commented 8 years ago

In theory we could support such concurrency using SharedArrayBuffer, but it's only in Firefox Nightly so far.