search

hygieia / hygieia

CapitalOne DevOps Dashboard
http://hygieia.github.io/hygieia
Apache License 2.0
3.79k stars 1.79k forks source link

Jira Collector fails when retrieving FeatureData #610

Closed mmullerbr closed 7 years ago

mmullerbr commented 8 years ago

Hi, guys!

I'm facing an error when running the collector: ERROR c.c.d.c.s.FeatureDataClientSetupImpl - Unexpected error in Jira paging request of java.lang.NullPointerException.

The project name is displayed in the Jira dashboard configuration dropbox, but no other information is presented after I select one project.

This is my collector log: 2016-06-01 13:44:00,016 INFO c.c.d.collector.FeatureCollectorTask - Starting Feature collection... 2016-06-01 13:44:00,054 INFO c.c.d.c.team.TeamDataClientSetupImpl - Beginning collection of team data at Wed Jun 01 13:44:00 UTC 2016 2016-06-01 13:44:01,143 INFO c.c.d.c.team.TeamDataClientSetupImpl - Process took :1.08712382 seconds to update 2016-06-01 13:44:01,150 INFO c.c.d.c.p.ProjectDataClientSetupImpl - Beginning collection of project data at Wed Jun 01 13:44:01 UTC 2016 2016-06-01 13:44:02,040 INFO c.c.d.c.p.ProjectDataClientSetupImpl - Process took :0.887490468 seconds to update 2016-06-01 13:44:02,045 INFO c.c.d.c.s.FeatureDataClientSetupImpl - Beginning collection of feature data at Wed Jun 01 13:44:02 UTC 2016 2016-06-01 13:44:02,960 ERROR c.c.d.c.s.FeatureDataClientSetupImpl - Unexpected error in Jira paging request of java.lang.NullPointerException [null]

This is my docker-compose Jira configuration: hygieia-jira-feature-collector: image: hygieia-jira-feature-collector:latest container_name: hygieia-jira volumes:

Thanks in advance,

tabladrum commented 8 years ago

@amp5208

amp5208 commented 8 years ago

@tabladrum: Question regarding the Dockerfile configurations - can I assume that your team has tested environment variable additions through a Docker container to the Feature Collectors? I ask because my code does not explicitly read system environment variable declarations, unless Java Springboot takes care of that for me.

@mmullerbr , Thank you for reaching out. Pending @tabladrum's answer above, please confirm the following parts of your configuration:

JIRA_BASE_URL=[your jira url, assuming that your Java keystore is updated with your SSL cert for jira]
JIRA_CREDENTIALS=[jira credentials with the following access format, in base64:  user:password]

# Also, you are missing your Jira field mappings to one of the following internal statuses:  todo, doing, done.  Examples:
feature.todoStatuses[0]=Open
feature.todoStatuses[1]=Groom
feature.todoStatuses[2]=Selected for Development
feature.todoStatuses[3]=Backlog
feature.todoStatuses[4]=Grooming
feature.todoStatuses[5]=ToDo
feature.todoStatuses[6]=To Do
feature.todoStatuses[6]=Backlog1
feature.doingStatuses[0]=Validation
feature.doingStatuses[1]=Test
feature.doingStatuses[2]=In Process
feature.doingStatuses[3]=In Progress
feature.doingStatuses[4]=Awaiting Approval - 2
feature.doingStatuses[5]=Change Request - 2
feature.doneStatuses[0]=Awaiting Approval
feature.doneStatuses[1]=Done
feature.doneStatuses[2]=Resolved

Please check the above configurations and let me know if you are still having issues.

tabladrum commented 8 years ago

@jimzucker - Can you see this if you have time? Thanks

jimzucker commented 8 years ago

@amp5208 @tabladrum Yes the Env variables set in the docker file & docker-compose files pass thru correctly. The spring classes are taking care of pulling them in and making them available internally. If you have a specific issue let us know.

amp5208 commented 8 years ago

@jimzucker , Thank you for confirming.

In that case, @mmullerbr , please see my updated and simplified application properties configuration details at the following README fork; try providing ALL of the properties that are relevant and not commented out via Docker ENVs: https://github.com/amp5208/Hygieia/blob/master/jira-feature-collector/README.md#sample-applicationproperties-file-with-minimum-overrides. I also provided explanations for most properties.

MarkRx commented 8 years ago

Just saw this as I was going to issues page. I was able to get the most recent jira collector working today (built off of master). I ran into a similar issue and found that I was missing some properties. Try comparing the properties that are generated in the docker container to this (my ordering is a little different):

#Database Name
database=dashboard

#Database HostName - default is localhost
dbhost=192.168.33.11

#Database Port - default is 27017
dbport=27017

#Database Username - default is blank
dbusername=db

dbpassword=dbpass

#Collector schedule (required)
feature.cron=0 0/5 * * * *

#Page size for data calls (Jira maxes at 1000)
feature.pageSize=1000

#In-built folder housing prepared REST queries (required)
feature.queryFolder=jiraapi-queries

#Jira API Query file names (String template requires the files to have .st extension) (required)
feature.storyQuery=story
feature.epicQuery=epic
feature.projectQuery=projectinfo
feature.memberQuery=memberinfo
feature.sprintQuery=sprintinfo
feature.teamQuery=teaminfo
feature.trendingQuery=trendinginfo

#Jira Connection Details
feature.jiraProxyUrl=
feature.jiraProxyPort=

# Trending Query:  Number of days in a sprint (not-required)
feature.sprintDays=60

# Trending Query:  Length of sprint week (not-required)
feature.sprintEndPrior=7

#Scheduled Job prior minutes to recover data created during execution time (usually, 2 minutes is enough)
feature.scheduledPriorMin=2

#Delta change date that modulates the collector item task - should be about as far back as possible, in ISO format (required)
feature.deltaCollectorItemStartDate=2008-01-01T00:00:00.000000

#Jira Connection Details
feature.jiraBaseUrl=<not shown>
feature.jiraQueryEndpoint=rest/api/2/

#reference:  https://www.base64decode.org/

#feature.jiraOauthAuthtoken=sdfghjkl==
#feature.jiraOauthRefreshtoken=sdfagheh==
#feature.jiraOauthRedirecturi=uri.this.is.test:uri
#feature.jiraOauthExpiretime=234567890987

#Start dates from which to begin collector data, if no other data is present - usually, a month back is appropriate (required)
feature.deltaStartDate=2015-03-01T00:00:00.000000
feature.masterStartDate=2008-01-01T00:00:00.000000

# In Jira, general IssueType IDs are associated to various "issue"
# attributes. However, there is one attribute which this collector's
# queries rely on that change between different instantiations of Jira.
# Please provide a numerical ID reference to your instance's IssueType for
# the lowest level of Issues (e.g., "user story") specific to your Jira
# instance.  Note:  You can retrieve your instance's IssueType ID
# listings via the following URI:  https://[your-jira-domain-name]/rest/api/2/issuetype/
feature.jiraIssueTypeId=6

# In Jira, your instance will have its own custom field created for "sprint" or "timebox" details,
# which includes a list of information.  This field allows you to specify that data field for your
# instance of Jira. Note: You can retrieve your instance's sprint data field name
# via the following URI, and look for a package name com.atlassian.greenhopper.service.sprint.Sprint;
# your custom field name describes the values in this field:
# https://[your-jira-domain-name]/rest/api/2/issue/[some-issue-name]
feature.jiraSprintDataFieldName=customfield_10005

# In Jira, your instance will have its own custom field created for "super story" or "epic" back-end ID,
# which includes a list of information.  This field allows you to specify that data field for your instance
# of Jira.  Note:  You can retrieve your instance's epic ID field name via the following URI where your
# queried user story issue has a super issue (e.g., epic) tied to it; your custom field name describes the
# epic value you expect to see, and is the only field that does this for a given issue:
# https://[your-jira-domain-name]/rest/api/2/issue/[some-issue-name]
feature.jiraEpicIdFieldName=customfield_10006

#Core Artifacts
# These arrays of status mappings must be completed in order for custom source system statuses to be providable by
# Hygieia.  Currently, Hygieia only maps to the following 3 status mappings:  to do, doing, and done.  You can find
# all available issue status mappings by going to your instance's version of the following API call:
# http://fake.jira.com/rest/api/2/status/.  Your statuses are listed in the JSON response as "name."
feature.todoStatuses[0]=OPEN
feature.doingStatuses[0]=IN PROGRESS
feature.doingStatuses[1]=TEST
feature.doingStatuses[2]=TESTING
feature.doneStatuses[0]=DONE
feature.doneStatuses[1]=CLOSED

feature.jiraCredentials=<not shown>

logging.level.com.capitalone.dashboard.collector=DEBUG
logging.level.com.atlassian.jira=INFO
#logging.level.org.apache.http.impl.nio.client.DefaultHttpAsyncClient=DEBUG
logging.level.com.capitalone.dashboard=DEBUG
#logging.level.org.springframework=DEBUG
amp5208 commented 8 years ago

@MarkRx , Thank you for following up regarding this issue. I also see some issues with your property configuration. See the following for changes:

feature.jiraIssueTypeId=6 # --> Should be a string value that corresponds to the given ID, e.g., "Story" or "Epic"

# Leave the following blanked-out
feature.jiraOauthAuthtoken=
feature.jiraOauthRefreshtoken=
feature.jiraOauthRedirecturi=
feature.jiraOauthExpiretime=

# Comment these lines out if you have no proxy:
#feature.jiraProxyUrl=
#feature.jiraProxyPort=

# Reduce your page size:
feature.pageSize=100

Essentially, make your property configs/environment settings for Docker match that of the property configs on this README: https://github.com/amp5208/Hygieia/blob/master/jira-feature-collector/README.md#sample-applicationproperties-file-with-minimum-overrides

MarkRx commented 8 years ago

@amp5208 I had originally commented out the proxy configuration but If I remember correctly that resulted in a NPE at FeatureDataClientSetupImpl:87:

    if (!this.featureSettings.getJiraProxyUrl().isEmpty()
mmullerbr commented 8 years ago

Hi, Guys! Thaks for the feedback!

@amp5208, I have tried adding the fields mapping before, but this time I realized that after a docker restart (in order to restart the collector and get the properties) all changes made manually in the properties file are wiped out. (Is there a way to add more properties via docker configuration or should I open another issue for this?)

In order to work around this, I had to stop the collector process (killing it forces a Docker stop/start and consequently the additional properties are wiped out) and run a new collector process (maybe I'm missing something, because I'm not a Linux expert).

By doing this the collector runs without errors.

nohup java -jar jira-feature-collector.jar --spring.config.location=hygieia-jira-feature-collector-new.properties &

2016-06-02 16:21:03,039 INFO c.c.d.collector.FeatureCollectorTask - Feature Data Collection Finished 2016-06-02 16:22:00,000 INFO c.c.d.collector.CollectorTask - Running Collector: Jira 2016-06-02 16:22:00,003 INFO c.c.d.collector.FeatureCollectorTask - Starting Feature collection... 2016-06-02 16:22:00,015 INFO c.c.d.c.team.TeamDataClientSetupImpl - Beginning collection of team data at Thu Jun 02 16:22:00 UTC 2016 2016-06-02 16:22:01,006 INFO c.c.d.c.team.TeamDataClientSetupImpl - Process took :0.990285674 seconds to update 2016-06-02 16:22:01,012 INFO c.c.d.c.p.ProjectDataClientSetupImpl - Beginning collection of project data at Thu Jun 02 16:22:01 UTC 2016 2016-06-02 16:22:01,979 INFO c.c.d.c.p.ProjectDataClientSetupImpl - Process took :0.966078813 seconds to update 2016-06-02 16:22:01,998 INFO c.c.d.c.s.FeatureDataClientSetupImpl - Beginning collection of feature data at Thu Jun 02 16:22:01 UTC 2016 2016-06-02 16:22:02,984 INFO c.c.d.c.s.FeatureDataClientSetupImpl - Process took :0.986238778 seconds to update 2016-06-02 16:22:02,984 INFO c.c.d.collector.FeatureCollectorTask - Feature Data Collection Finished

However, no data is being presented on the dashboard. Here follows my new properties file:

Database

database=dashboard dbhost=172.17.0.2 dbport=27017 dbusername=db dbpassword=dbpass

feature.pageSize=100

feature.deltaStartDate=2016-03-01T00:00:00.000000 feature.masterStartDate=2016-03-01T00:00:00.000000 feature.deltaCollectorItemStartDate=2016-03-01T00:00:00.000000

feature.cron=0 * * * * *

feature.queryFolder=jiraapi-queries feature.storyQuery=story feature.epicQuery=epic

No proxy

feature.jiraProxyUrl= feature.jiraProxyPort=

feature.jiraBaseUrl=https://myJira.atlassian.net feature.jiraQueryEndpoint=rest/api/2/ feature.jiraCredentials=asasasasasasasasasas

OAuth is not fully implemented; please blank-out the OAuth values

feature.jiraOauthAuthtoken= feature.jiraOauthRefreshtoken= feature.jiraOauthRedirecturi= feature.jiraOauthExpiretime=

feature.jiraIssueTypeId=Story feature.jiraSprintDataFieldName=customfield_10017 feature.jiraEpicIdFieldName=customfield_10018

feature.todoStatuses[0]=To Do feature.doingStatuses[0]=In Progress feature.doneStatuses[0]=Done

[{...name:"In Progress"...},{... name: "Done" ...},{... name: "To Do" ...}]

Thanks in advance,

amp5208 commented 8 years ago

@MarkRx : Did you receive this NullPointerException with the newest version on upstream master? I ask because issues related to the proxy configuration were fixed in a previous issue/PR (https://github.com/capitalone/Hygieia/pull/322)

@mmullerbr : There is, in fact, a different way to pass in the property file for Hygieia. You may use the Docker -v [dockerhost dir]:[dockercontainer dir] to volume map to a local property file configuration from within the Docker container; more details can be found on the Docker official content page: Volume Mapping. Additionally, you may have to force a different CMD run option for the Docker container, so that it knows where to find your application.properties file; this can be done by adding a new java command at run time at the end of the docker run command: java -jar jira-feature-collector.jar --spring.config.location=hygieia-jira-feature-collector-new.properties. Please see below for a sample command / setup:

#!/bin/bash
docker run -d -p 8080:8080 -v $SRC_DIR/hygieia-jira-feature-collector-new.properties:$DOCKER_CNTR_DIR/hygieia-jira-feature-collector-new.properties $HYGIEIA_DOCKER_IMAGE_TAG:latest kill $(pgrep -o java) && java -jar jira-feature-collector.jar --spring.config.location=$DOCKER_CNTR_DIR/hygieia-jira-feature-collector-new.properties

@jimzucker : Please verify my above statements and make corrections as needed.

MarkRx commented 8 years ago

I just submitted PR #614 will make using external custom configs easier.

@amp5208 Yes. Here is the stack trace:

2016-06-03 15:06:00,067 ERROR o.s.s.s.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
java.lang.NullPointerException: null
        at com.capitalone.dashboard.client.team.TeamDataClientSetupImpl.updateObjectInformation(TeamDataClientSetupImpl.java:85) ~[jira-feature-collector.jar!/:2.0.2-SNAPSHOT]
        at com.capitalone.dashboard.client.team.TeamDataClientImpl.updateTeamInformation(TeamDataClientImpl.java:130) ~[jira-feature-collector.jar!/:2.0.2-SNAPSHOT]
        at com.capitalone.dashboard.collector.FeatureCollectorTask.collect(FeatureCollectorTask.java:99) ~[jira-feature-collector.jar!/:2.0.2-SNAPSHOT]
        at com.capitalone.dashboard.collector.FeatureCollectorTask.collect(FeatureCollectorTask.java:26) ~[jira-feature-collector.jar!/:2.0.2-SNAPSHOT]
        at com.capitalone.dashboard.collector.CollectorTask.run(CollectorTask.java:63) ~[core-2.0.2-SNAPSHOT.jar!/:2.0.2-SNAPSHOT]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81) [spring-context-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_74]
        at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_74]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) [na:1.8.0_74]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.8.0_74]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_74]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_74]
        at java.lang.Thread.run(Unknown Source) [na:1.8.0_74]
jimzucker commented 8 years ago

@mmullerbrhttps://github.com/mmullerbr - The standard docker pattern is to create a docker-compose.override.yml and change/define whatever you want there. This will change what is defined in the config file after you run docker compose.

Here is the context of my file that I used for testing:

hygieia-github-scm-collector: environment:

amp5208 commented 8 years ago

@MarkRx , Thank you for following up with the stack trace, and thank you @jimzucker for the Docker compose expertise.

It seems that you are receiving the error likely because the proxy setting variables are, in fact null, not blank. the line of code affected here is this: https://github.com/capitalone/Hygieia/blob/master/jira-feature-collector/src/main/java/com/capitalone/dashboard/client/team/TeamDataClientSetupImpl.java#L85. I will check this on my next commit and submit a PR if needed. For now, setting proxy to blank. i.e.: 

feature.jiraProxyUrl=
feature.jiraProxyPort=

... should do the trick. Let me know if you are still seeing odd behavior.

mmullerbr commented 8 years ago

@jimzucker, the jira-properties-builder.sh is not prepared to receive field mapping form docker-compose file and replace them in the properties file. I was able to modify the jira-properties-builder.sh inside the container and get the collector up and running with the properties file presented below. However I will loose my changes every time a container recreation is needed - this is a problem for another day... :)

Unfortunately my dash board remains not presenting sprints and tasks information. Any other thoughts?

Database

database=dashboard dbhost=172.17.0.2 dbport=27017 dbusername=db dbpassword=dbpass

feature.cron=0 * * * * * feature.pageSize=100

feature.queryFolder=jiraapi-queries feature.storyQuery=story feature.epicQuery=epic

Jira Connection Details

feature.jiraProxyUrl= feature.jiraProxyPort=

feature.jiraBaseUrl=https://myJira.atlassian.net/ feature.jiraQueryEndpoint=rest/api/2/ feature.jiraCredentials=asasasasasasasa

OAuth2.0 token credentials (currently not supported in this version)

feature.jiraOauthAuthtoken= feature.jiraOauthRefreshtoken= feature.jiraOauthRedirecturi= feature.jiraOauthExpiretime=

feature.deltaCollectorItemStartDate=2016-01-01T00:00:00.000000 feature.deltaStartDate=2016-01-01T00:00:00.000000 feature.masterStartDate=2016-01-01T00:00:00.000000

feature.jiraIssueTypeId=Story feature.jiraSprintDataFieldName=customfield_10017 feature.jiraEpicIdFieldName=customfield_10018

feature.todoStatuses[0]=To Do feature.doingStatuses[0]=In Progress feature.doneStatuses[0]=Done

[{...name:"In Progress"...},{... name: "Done" ...},{... name: "To Do" ...}]

jimzucker commented 8 years ago

@mmullerbr I am not clear on the problem can you give me a simple example?

amp5208 commented 8 years ago

@mmullerbr , At this point, I will need some stack traces and/or UI console logs for further debug, as your property configurations appear correct at face-value. Please provide those at your convenience.

satishc1 commented 7 years ago

Closed. Reopen if there are any problems. Use latest codebase and follow cleanup instructions provided in https://github.com/capitalone/Hygieia/pull/1137.