search

jhurliman / node-echoprint-server

A node.js implementation of the Echoprint music identification server
74 stars 37 forks source link

Ingest problems... #9

Closed onosendi closed 2 years ago

onosendi commented 11 years ago

So, after almost two days of trying to debug this issue, I thought it may be time to ask you. Let me start off by saying, your work is awesome, and greatly appreciated... It's so much easier working with JavaScript/MySQL than running tyrants from tokyo etc. :P

My server is running off of a custom build based on Arch Linux. I am using echoprint-codegen as follows: echoprint-codegen filename.mp3 10 40 on a 60 second track.

To ingest, I'm using: curl http://localhost:10090/ingest -d "code=&length=30&version=4.12"

The first error I ran into, was constant "Missing required fields error." Therefore, I commented out "if (!fp.codes.length || typeof fp.length !== 'number' || !fp.codever || !fp.track || !fp.artist) return callback('Missing required fields', null);" to get past it, in which it succeeded.

The next error I run into. Everything was being inserted into the mysql database, except for the code, and I was getting a "access denied for blah@'%' using password (yes)" error from the mysql database. This was fixed by using the root username/password for the database. So in other words, it had access to write everything to the sql database except for the code.

So after that was fixed, I ran into my next problem. I kept getting a "Cannot stat echoprint-?-?.csv" I fixed this by changing 'LOAD DATA INFILE ?..." to "LOAD DATA LOCAL INFILE" in 'models/mysql.js'.

Now with the error messages out of the way, the code is still not being inserted into the database. After going through pretty much every file, I can't find anywhere where it queries the mysql database, and inserts the 'code' section. I see "INSERT INTO artists (name) etc, but nothing saying to write the code to the "codes" table.

Summary: without "if (!fp.codes.length || typeof..." commented out, I get 'missing required fields' error. Second, the 'code' generated from echoprint-codegen isn't being inserted into the database. I'm guessing these two issues are related? Any help would be greatly appreciated, thank you.

MattNewberry commented 11 years ago

I haven't resolved all of the related issues yet, however after stumbling upon the same as you, i found that

database.addTrack(fp, function(err, trackID) is invalid

The method is function addTrack(artistID, fp, callback)

thus, the artistID is not being passed when it's called.

MattNewberry commented 11 years ago

To add, the failure comes with ingestion not adding a new artist (or looking up an existing one). Passing the id of an existing artist to database.addTrack resolves the issue and ingests properly.

jhurliman commented 11 years ago

@MattNewberry are you sure you're running the latest version of the code? At https://github.com/jhurliman/node-echoprint-server/blob/master/controllers/fingerprinter.js#L425 you can see the call is database.addTrack(artistID, fp, function(err, trackID) { ... });

MattNewberry commented 11 years ago

Arg, apparently not! Apologies

jhurliman commented 11 years ago

@onosendi I just committed a fix to print out more descriptive error messages when the metadata is incomplete. I haven't done a good job of explaining this in the documentation, but when ingesting you need to fill in the "track" and "artist" fields in the JSON output from echoprint-codegen. Otherwise, the fingerprinting server has no idea what track you're ingesting. It would be nice if echoprint-codegen could automatically pull this information out of MP3 metadata, but right now it is just operating on the raw waveform of audio files and ignoring any metadata.

The database.addTrack() method writes all of the codes to a temporary text file and uses MySQL's "LOAD DATA LOCAL INFILE" to do a bulk insert. This is called from fingerprinter.js createTrack(). The most likely explanation is that the artist and track fields were not filled in which created problems elsewhere.

jhurliman commented 11 years ago

@onosendi @MattNewberry I just made a series of commits that should fix ingestion issues and return better error messages when something goes wrong. Please pull the latest version, run npm install again, and let me know if everything is working now. You might also want to drop data from your current echoprint MySQL table to start with a clean slate.

MattNewberry commented 11 years ago

@jhurliman Much obliged! Was just writing you about the mysql lock, perfect timing!

mrrad commented 11 years ago

I believe there is a related issue with api.ingest. api.ingest fails if length is a string because fingerprint.ingest fails if length is a string and api.ingest does not convert it. I suggest changing:

controllers/api.js:50 var length = req.body.length;

to

controllers/api.js:50 var length = parseInt(req.body.length, 10);

and

controllers/api.js:60 if (isNaN(parseInt(length, 10)))

to

controllers/api.js:60 if (isNaN(length))

masbog commented 11 years ago

@mrrad it's work, but i have problem in ingest some music, i have result from console terminal

error: Timed out while responding to a request from (ip address),

so i check in server.js at line 18 : var TIMEOUT = 1000 * 60; and then change it to biggest number of timeout, but i have some error result

somebody can resolve it?

chilinhnet commented 10 years ago

very slow with mysql, always timeout finger and ingest

onosendi commented 10 years ago

@MattNewberry Hey man, I just wanted to thank you for getting back to me, I didn't see this until now. I ended up writing an api in Python, with tyrant as the database. When I'm finished with this project I'll play around with this again. Is everything working as it should?

ChandraManiSingh commented 10 years ago

Hi there, I am facing "Time out" issue while ingesting code in database. Please help.

Below is log -

debug: Inflated 5330 byte code string into 1248 codes info: Ingesting track "Besharmi_Ki_Height" by artist "Unknown", 40 seconds, 1248 codes (4.12) debug: Starting query with 1248 codes debug: Matched 1 tracks, top code overlap is 96 debug: Track does not exist in the database yet, status MULTIPLE_BAD_HISTOGRAM_M ATCH debug: Adding track "Besharmi_Ki_Height" for artist "Unknown" (34) debug: Writing 1248 codes to temporary file C:\Users\CHANDR~1.SIN\AppData\Local\ Temp\echoprint-23114325-7960-f4a3j6.csv debug: Bulk inserting 1248 codes for track 23 error: Timed out while responding to a request from 127.0.0.1

slusiola commented 9 years ago

I have over 10K tracks. How do I do a bulk ingest?

I have successfully installed the system, I can ingest via myserver:37760/debug, however, I need to figure out a way to script it. I've tried using curl but I still end up 'Missing a field'... I would be grateful if someone shared with me a working script from bulk ingestion.

andresetevejob commented 8 years ago

Hi All I have this error when i test a soung from my android to node echoprintserver : "No match code "But it's the same song with the song server