neume-network extraction-worker issues

neume-network / extraction-worker

Async worker_thread component for maximalizing concurrent data retrieval and processing.

GNU General Public License v3.0

3 stars 5 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

allow extraction-worker to be used without worker_thread

#60 il3ven closed 1 year ago
2
debug doesn't show logs for validation with DEBUG=* in /core

#59 TimDaub opened 2 years ago
0
ipfs request with header = null won't work for rapidly requesting gateway endpoints

#58 TimDaub closed 2 years ago
0
update @neume-network/schema@0.8.0

#57 TimDaub opened 2 years ago
0
extraction worker still uses message-schema, but it has been deprecated

#56 TimDaub closed 2 years ago
2
feat/add-arweave-api

#55 Kunkka0822 closed 2 years ago
5
try to parse all requests as json

#54 il3ven closed 2 years ago
1
test cases for result json parsing

#53 TimDaub closed 2 years ago
0
Implement Arweave support

#52 TimDaub closed 2 years ago
4
Implement content negotiation using `Accept` and `Content-Type` header for worker

#51 TimDaub closed 2 years ago
1
parse response as json only if content-type includes json

#50 il3ven closed 2 years ago
0
extraction-worker shouldn't try to `JSON.parse` all responses

#49 il3ven closed 2 years ago
0
Use weiroll to batch process multiple eth_call requests in a single RPC call

#48 TimDaub closed 2 years ago
1
feat: handle ipfs messages

#47 il3ven closed 2 years ago
5
Basing extraction-worker on nodejs disallows usage in the browser

#46 TimDaub opened 2 years ago
1
for any IPFS URI, we're currently not validating if the request's results are matching the hash

#45 TimDaub opened 2 years ago
0
validate config before running

#44 il3ven closed 2 years ago
3
fastq implementation

#43 TimDaub closed 2 years ago
1
create a set of criteria for what we need in a queue. Survey js ecosystem for the best npm package

#42 TimDaub closed 2 years ago
1
switch out better-queue with e.g. fastq to understand if queue causes sudden staling

#41 TimDaub closed 2 years ago
0
call-block-logs makes worker run out of heap memory

#40 TimDaub opened 2 years ago
0
`workerData.endpoints` should use neume-network/schema

#39 TimDaub closed 2 years ago
3
add rate limiter

#38 il3ven closed 2 years ago
1
Return message object from panic function

#37 il3ven closed 2 years ago
0
Rate limits should be definable by extraction-worker pool

#36 TimDaub closed 2 years ago
1
Tests fails because setTimeout cannot be set in expressively-mocked-fetch

#35 TimDaub closed 2 years ago
0
panic returns context free error in false message object

#34 TimDaub closed 2 years ago
2
some blatantly wrong IPFS CIDs aren't immediately purged from the worker

#33 TimDaub closed 2 years ago
1
current implementation of AbortSignal using `setTimeout` sequentializes extraction worker (blocking call)

#32 TimDaub closed 2 years ago
2
extraction-worker implements `setTimeout` falsely

#31 TimDaub closed 2 years ago
0
STATUS 429 has a Retry-After

#30 TimDaub opened 2 years ago
2
We're not distinguishing between permanent or temporary request failures (e.g. for permanently deleted IPFS CIDs)

#29 TimDaub opened 2 years ago
0
current queue usage doesn't batch tasks

#28 TimDaub closed 2 years ago
2
there are cases in extraction-worker where only a string can be returned as the result of a message object

#27 TimDaub closed 2 years ago
1
log to signal eth_call is not useful and leads to the debug log being spammed

#26 TimDaub closed 2 years ago
0
We must allow a user to pass better-queue options from outside

#25 TimDaub closed 2 years ago
0
function panic to handle better-queue errors is assuming false callback structure

#24 TimDaub closed 2 years ago
0
It's difficult to understand if a very long timeout in `fetch` could stall the extraction worker's progress

#23 TimDaub closed 2 years ago
5
It's hard to debug how many requests per second extraction worker does

#22 TimDaub closed 2 years ago
1
error message for failing HTTPS request isn't useful for debugging

#20 TimDaub closed 2 years ago
0
some requests to soundxyz's API fail, are we rate limited?

#21 TimDaub opened 2 years ago
3
extraction-worker should parallelize different root domains

#19 TimDaub opened 2 years ago
7
Implement getLogs

#18 TimDaub closed 2 years ago
0
if a string blockNumber "arbitrary" is passed to "eth_call" it can be that simply no results are returned. `results: undefined`. It should throw

#17 TimDaub closed 2 years ago
6
must define eth-fun@0.7.1 as peerDependency

#16 TimDaub closed 2 years ago
0
make `concurrency` option in queue configurable with env vars

#15 TimDaub closed 2 years ago
1
allow downloading data from ipfs://

#14 TimDaub closed 2 years ago
0
upgrade eth-fun@0.7.1

#13 TimDaub closed 2 years ago
1
update message-schema version, fix unit tests and fix prettier CI

#12 il3ven closed 2 years ago
2
on npm, release extraction worker under new organization scope

#11 TimDaub closed 2 years ago
0