How can I extract a regional species list (less than 984) if there is no eBird information in my area?

lenawarbler commented 3 years ago

Hi birdNET fans!

I have birdNET running on Windows and I am really impressed!! It is 'mostly' detecting the correct species in the Yukon, Canada, HOWEVER, birdNET is not reducing the candidate species list (from 984) and is reporting many European and southern species that do not occur here.

I think the issue is: there is little/no eBird information this far north so birdNET cannot generate a candidate species list and defaults to 984 species. When I specify coordinates from populated areas in Canada (i.e., Vancouver BC) it filters to 160 or so species, but the detections are still not quite right (e.g., birdNET filters out northern species like ptarmigans etc). I tried specifying coordinates for northern cities like Whitehorse YT or Anchorage AK where there are far more eBird checklists, but that did not extract a species list (not enough checklists?).

I have tried to manually change GRID_STEP_SIZE in config.py to see if I could extract information from a larger spatial area (I think it uses the 3x3km eBird grid cells?). Again, this worked for the southern areas (I increased the species pool in Sapsucker Woods from 130 to 180) BUT ALAS it still does not work in Northern regions.

What should I do? Can I manually feed birdNET my own species list? Or can I alter the python code do draw checklists from a much larger area? ie Yukon/Alaska/NWT should have enough information??

Thank you for helping!

Update: @kahst please help!

abfleishman commented 3 years ago

I have a similar issue. We are trying to run birdnet on many sites close to each other and SHOULD all have the same species list. But the eBird data are spotty and each site gets a different species list. We would like to supply our own list based on our species of interest. @lenawarbler did you ever work through this? @kahst Any thoughts on where we should start?

lenawarbler commented 3 years ago

Hi @abfleishman! Yes, being able to supply our own species lists would be ideal. I have emailed the Cornell crew about this and have had trouble getting through to anyone for help. Please let me know if you find a work around.

ghost commented 3 years ago

Folks,

I understand and appreciate the intentions of the good folks who produced all of this code for everyone to use. But I also recognized the need to limit detection matches to a set of species that have ever been detected locally. I altered the code to restrict detections to a certain list.

In config.py, I introduce: HSA_SPECIES_CODES = 'metadata/HSA.json which is a JSON list of all the species ever recorded in the Hamilton Study Area.

I then modified metadata/grid.py like so: (base) @.***:[~/BirdNET_DLW/metadata]$ diff grid.py grid.py.ORIG 16d15 < HSACODES = [] 24,27d22 < global HSACODES < global HSAscicom < global HSAcom < global HSAsci 47,58d41 < # Load HSA codes < with open(cfg.HSA_SPECIESCODES, 'r') as jfile2: < HSACODES = json.load(jfile2) < < HSAscicom = list(HSACODES.values()) < HSAcom = [] < HSAsci = [] < for b in HSAscicom: < (sci, com) = b.split('') < HSAsci.append(sci) < HSAcom.append(com) < 93,99c76,110 < #--DLW This looks like a good place to restrict to HSA species! < for i in range(len(cfg.CLASSES)): < if cfg.CLASSES[i].split('_')[0] in HSAsci: < p[i] = 1.0 < else: < p[i] = 0.0 < return p

         return p + 1.0      else:

         # Get checklist data for nearest cell          cdata = getCellData(lat, lon)

         # No cell data?          if cdata == None:              return p + 1.0          else:

             # Get probabilities from checklist frequencies              for entry in cdata['data']:                  for species in entry:

                     try:                          # Get class index from species code                          for i in range(len(cfg.CLASSES)):                              if cfg.CLASSES[i].split('')[0] == CODES[species].split('')[0]:

                                 # Do we want a specific week?                                  if week >= 1 and week <= 48:                                      p[i] = entry[species][week - 1] / 100.0

                                 # If not, simply return the max frequency                                  else:                                      p[i] = max(entry[species]) / 100.0

                                 break

                     except:                          pass

         return p

The upshot is no more Flammulated Owl detections and no constraints due to eBird reporting limitations. The purposes of the monitoring done by our recorders is to identify transient/rare species that might not be detected at all if a 2% percent threshold on eBird reports in a grid cell in a given week is a thing.

Cheers, Doug

On 10/6/21 6:21 PM, lenawarbler wrote:

Hi @abfleishman https://github.com/abfleishman! Yes, being able to supply our own species lists would be ideal. I have emailed the Cornell crew about this and have had trouble getting through to anyone for help. Please let me know if you find a work around.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kahst/BirdNET/issues/15#issuecomment-937255700, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABIZZ3VTUB66K25RN4YSQ23UFTDWRANCNFSM46RVIPQQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

mcguirepr89 commented 3 years ago

Hello, all -- I just wanted to point out the BirdNET-Lite repo, which already supports custom species lists and +6,000 species worldwide, for those who need more species. My best, Patrick

kahst / BirdNET

How can I extract a regional species list (less than 984) if there is no eBird information in my area? #15